00010000:00010000:2.0F:1777502488.017670:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0F:1777502489.041687:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0F:1777502489.244694:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502489.244738:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502489.244741:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567206400 00000100:00100000:2.0:1777502489.244751:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567206400 00000100:00100000:3.0:1777502489.244769:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502489.244800:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502489.244809:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 379 00000100:00100000:2.0:1777502489.244823:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45cce050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:c3865704-aa9b-444e-8420-b5af90b3c217+9:2373:x1863837567206400:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:3.0:1777502489.244835:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567206528 00000100:00100000:3.0:1777502489.244840:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567206528 00000100:00100000:2.0:1777502489.244889:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45cce050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:c3865704-aa9b-444e-8420-b5af90b3c217+9:2373:x1863837567206400:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 69us (198us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502489.244895:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 442 00000100:00100000:2.0:1777502489.244902:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 379 00000100:00100000:3.0:1777502489.244908:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc547b2300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0775d2ab-558e-41f1-82c0-14244d133390+15:2372:x1863837567206528:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:3.0:1777502489.244949:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc547b2300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0775d2ab-558e-41f1-82c0-14244d133390+15:2372:x1863837567206528:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 45us (209us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502489.244958:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 442 00000100:00100000:3.0:1777502489.244989:0:205386:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567206784 00000100:00100000:3.0:1777502489.244994:0:205386:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567206784 00000100:00100000:3.0:1777502489.245033:0:205386:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567206656 00000100:00100000:3.0:1777502489.245037:0:205386:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567206656 00000100:00100000:2.0:1777502489.245059:0:205380:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 466 00000100:00100000:2.0:1777502489.245069:0:205380:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc547b0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_012:0775d2ab-558e-41f1-82c0-14244d133390+7:2372:x1863837567206784:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:3.0:1777502489.245083:0:205386:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 467 00000100:00100000:2.0:1777502489.245110:0:205380:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc547b0000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_012:0775d2ab-558e-41f1-82c0-14244d133390+7:2372:x1863837567206784:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 43us (340us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502489.245147:0:205380:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 466 00000100:00100000:3.0:1777502489.245153:0:205386:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc547b0a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_018:0775d2ab-558e-41f1-82c0-14244d133390+6:2373:x1863837567206656:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:3.0:1777502489.245187:0:205386:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc547b0a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_018:0775d2ab-558e-41f1-82c0-14244d133390+6:2373:x1863837567206656:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 39us (388us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502489.245195:0:205386:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 467 00000100:00080000:3.0:1777502489.245222:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:3.0:1777502489.245227:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:3.0:1777502489.245230:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:0.0:1777502489.681725:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4479ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3311:1863837562073984:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1777502489.681791:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502489.682331:0:187552:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562073984 00000100:00100000:2.0:1777502489.682339:0:187552:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562073984 00000100:00100000:3.0:1777502489.682360:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc547b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562074240:0@lo:400:kworker.0 00000100:00100000:2.0:1777502489.682417:0:187552:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 260 00000100:00100000:2.0:1777502489.682430:0:187552:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc44799500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3311:x1863837562073984:12345-0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1777502489.682452:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc547b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3310:1863837562074368:0@lo:400:kworker.0 00002000:00000024:2.0:1777502489.682467:0:187552:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 42008576 pending 0 free 3831496704 avail 3829399552 00000100:00100000:3.0:1777502489.682486:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:3.0:1777502489.682527:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1777502489.682528:0:187552:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc44799500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3311:x1863837562073984:12345-0@lo:13:osp-pre-0-0.0 Request processed in 101us (735us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502489.682537:0:187552:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 260 00000100:00100000:3.0:1777502489.682545:0:205379:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562074368 00000100:00100000:3.0:1777502489.682553:0:205379:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562074368 00000100:00100000:3.0:1777502489.682612:0:205379:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 468 00000100:00100000:3.0:1777502489.682619:0:205379:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc547b2680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_011:lustre-MDT0000-mdtlov_UUID+5:3310:x1863837562074368:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502489.682663:0:205379:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc547b2680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_011:lustre-MDT0000-mdtlov_UUID+5:3310:x1863837562074368:12345-0@lo:400:kworker.0 Request processed in 44us (178us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502489.682672:0:205379:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 468 00000100:00080000:3.0:1777502489.682695:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:3.0:1777502489.682729:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502489.682782:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc547b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3309:1863837562074752:0@lo:400:kworker.0 00000100:00100000:3.0:1777502489.682810:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502489.682854:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562074752 00000100:00100000:3.0:1777502489.682857:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562074752 00000100:00100000:3.0:1777502489.682898:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 443 00000100:00100000:3.0:1777502489.682906:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc547b1500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0001_UUID+5:3309:x1863837562074752:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502489.682942:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562074240 00000100:00100000:3.0:1777502489.682945:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc547b1500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0001_UUID+5:3309:x1863837562074752:12345-0@lo:400:kworker.0 Request processed in 40us (135us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502489.682947:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562074240 00000100:00100000:3.0:1777502489.682954:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 443 00000100:00080000:3.0:1777502489.682983:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:2.0:1777502489.682986:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 380 00000100:00100000:2.0:1777502489.682998:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45cccc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3309:x1863837562074240:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502489.683016:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc547b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3310:1863837562074368:0@lo:400:kworker.0 00000100:00100000:2.0:1777502489.683060:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45cccc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3309:x1863837562074240:12345-0@lo:400:kworker.0 Request processed in 65us (332us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502489.683070:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc547b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562074240:0@lo:400:kworker.0 00000100:00100000:2.0:1777502489.683080:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 380 00000100:00100000:3.0:1777502489.683094:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc547b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0001_UUID:3309:1863837562074752:0@lo:400:kworker.0 00000100:00100000:2.0:1777502489.683123:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562074112:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1777502489.683147:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502489.683175:0:187553:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562074112 00000100:00100000:2.0:1777502489.683176:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc547b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3312:1863837562074624:0@lo:400:kworker.0 00000100:00100000:0.0:1777502489.683182:0:187553:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562074112 00000100:00100000:2.0:1777502489.683210:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502489.683224:0:187553:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 261 00000100:00100000:0.0:1777502489.683234:0:187553:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493f8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562074112:12345-0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1777502489.683243:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:2.0:1777502489.683253:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc547b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0000_UUID:3312:1863837562074496:0@lo:400:kworker.0 00002000:00000024:0.0:1777502489.683254:0:187553:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 43712512 pending 0 free 3801088000 avail 3798990848 00000100:00100000:2.0:1777502489.683293:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502489.683298:0:187553:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493f8e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562074112:12345-0@lo:13:osp-pre-1-0.0 Request processed in 65us (149us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502489.683304:0:187553:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 261 00000100:00100000:2.0:1777502489.683333:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562074112:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1777502489.683371:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562074624 00000100:00100000:2.0:1777502489.683375:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562074624 00000100:00100000:2.0:1777502489.683423:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562074496 00000100:00100000:2.0:1777502489.683426:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562074496 00000100:00100000:2.0:1777502489.683463:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 444 00000100:00100000:2.0:1777502489.683472:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493fad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0000_UUID+5:3312:x1863837562074624:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502489.683520:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493fad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0000_UUID+5:3312:x1863837562074624:12345-0@lo:400:kworker.0 Request processed in 51us (310us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502489.683529:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 444 00000100:00100000:2.0:1777502489.683549:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 445 00000100:00100000:2.0:1777502489.683558:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc56069f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0000_UUID+5:3312:x1863837562074496:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502489.683713:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc56069f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0000_UUID+5:3312:x1863837562074496:12345-0@lo:400:kworker.0 Request processed in 156us (419us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502489.683721:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 445 00000100:00080000:2.0:1777502489.683755:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:2.0:1777502489.683792:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc547b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0000_UUID:3312:1863837562074496:0@lo:400:kworker.0 00000100:00100000:2.0:1777502489.683815:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc547b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3312:1863837562074624:0@lo:400:kworker.0 00000100:00100000:2.0:1777502489.683863:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4479ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3311:1863837562073984:0@lo:13:osp-pre-0-0.0 00010000:00010000:0.0:1777502490.065722:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502491.089667:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1777502491.337454:0:220478:0:(debug.c:645:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanityn test 200: service remains healthy while able to process request ========================================================== 18:41:30 (1777502490) 00010000:00010000:2.0:1777502492.113664:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502493.137678:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502494.161726:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1777502494.364626:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502494.364663:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502494.364688:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567206912 00000100:00100000:2.0:1777502494.364696:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567206912 00000100:00100000:3.0:1777502494.364702:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502494.364739:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:0.0:1777502494.364780:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567207040 00000100:00100000:0.0:1777502494.364786:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567207040 00000100:00100000:3.0:1777502494.364790:0:187548:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567207168 00000100:00100000:2.0:1777502494.364798:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 381 00000100:00100000:3.0:1777502494.364806:0:187548:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567207168 00000100:00100000:2.0:1777502494.364814:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45ccf450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:c3865704-aa9b-444e-8420-b5af90b3c217+9:2373:x1863837567206912:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:3.0:1777502494.364844:0:187548:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567207296 00000100:00100000:3.0:1777502494.364849:0:187548:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567207296 00000100:00100000:0.0:1777502494.364873:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 446 00000100:00100000:0.0:1777502494.364883:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc547b2a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0775d2ab-558e-41f1-82c0-14244d133390+15:2372:x1863837567207040:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:3.0:1777502494.364888:0:187548:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 469 00000100:00100000:3.0:1777502494.364903:0:187548:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc547b0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:0775d2ab-558e-41f1-82c0-14244d133390+6:2373:x1863837567207168:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:2.0:1777502494.364905:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45ccf450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:c3865704-aa9b-444e-8420-b5af90b3c217+9:2373:x1863837567206912:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 93us (283us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502494.364918:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 381 00000100:00100000:0.0:1777502494.364920:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc547b2a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0775d2ab-558e-41f1-82c0-14244d133390+15:2372:x1863837567207040:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 38us (255us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502494.364927:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 446 00000100:00100000:3.0:1777502494.364941:0:187548:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc547b0e00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:0775d2ab-558e-41f1-82c0-14244d133390+6:2373:x1863837567207168:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 44us (240us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502494.364948:0:187548:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 469 00000100:00100000:3.0:1777502494.364968:0:187548:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 470 00000100:00100000:3.0:1777502494.364974:0:187548:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc547b0380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:0775d2ab-558e-41f1-82c0-14244d133390+7:2372:x1863837567207296:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:3.0:1777502494.365018:0:187548:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc547b0380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:0775d2ab-558e-41f1-82c0-14244d133390+7:2372:x1863837567207296:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 44us (280us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502494.365024:0:187548:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 470 00000100:00080000:2.0:1777502494.365055:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:2.0:1777502494.365061:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:2.0:1777502494.365066:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:0.0:1777502494.801780:0:3309:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1777502494.801804:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3309:1863837562075008:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1777502494.801804:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4479a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3311:1863837562074880:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1777502494.801908:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502494.802111:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4479aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:3310:1863837562075392:0@lo:400:kworker.0 00000100:00100000:0.0:1777502494.802162:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502494.802233:0:187552:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562075008 00000100:00100000:2.0:1777502494.802241:0:187552:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562075008 00000100:00100000:2.0:1777502494.802308:0:187552:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 262 00000100:00100000:2.0:1777502494.802321:0:187552:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc44798000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3309:x1863837562075008:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:2.0:1777502494.802357:0:187552:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 43712512 pending 0 free 3801088000 avail 3798990848 00000100:00100000:2.0:1777502494.802404:0:187552:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc44798000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3309:x1863837562075008:12345-0@lo:13:osp-pre-1-0.0 Request processed in 87us (497us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502494.802410:0:187552:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 262 00000100:00100000:0.0:1777502494.802439:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562075392 00000100:00100000:2.0:1777502494.802441:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4479b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562075136:0@lo:400:kworker.0 00000100:00100000:0.0:1777502494.802444:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562075392 00000100:00100000:2.0:1777502494.802492:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502494.802529:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 447 00000100:00100000:0.0:1777502494.802540:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4479ad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:lustre-MDT0000-lwp-MDT0000_UUID+5:3310:x1863837562075392:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502494.802541:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44798700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3312:1863837562075648:0@lo:400:kworker.0 00000100:00100000:2.0:1777502494.802598:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502494.802621:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4479ad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:lustre-MDT0000-lwp-MDT0000_UUID+5:3310:x1863837562075392:12345-0@lo:400:kworker.0 Request processed in 82us (459us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502494.802626:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502494.802628:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:0.0:1777502494.802637:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 447 00000100:00100000:2.0:1777502494.802638:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3312:1863837562075520:0@lo:400:kworker.0 00000100:00100000:0.0:1777502494.802663:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562075648 00000100:00100000:2.0:1777502494.802667:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502494.802669:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562075648 00000100:00100000:0.0:1777502494.802703:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562075520 00000100:00100000:0.0:1777502494.802707:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562075520 00000100:00100000:2.0:1777502494.802720:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562075136 00000100:00100000:2.0:1777502494.802724:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562075136 00000100:00100000:0.0:1777502494.802740:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 448 00000100:00100000:0.0:1777502494.802750:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f711500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:lustre-MDT0000-lwp-OST0001_UUID+5:3312:x1863837562075648:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502494.802760:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 382 00000100:00100000:3.0:1777502494.802772:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 449 00000100:00100000:2.0:1777502494.802773:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45ccc050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3312:x1863837562075136:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502494.802784:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd63da6680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0000_UUID+5:3312:x1863837562075520:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502494.802799:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f711500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:lustre-MDT0000-lwp-OST0001_UUID+5:3312:x1863837562075648:12345-0@lo:400:kworker.0 Request processed in 50us (199us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502494.802808:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 448 00000100:00100000:2.0:1777502494.802820:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45ccc050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3312:x1863837562075136:12345-0@lo:400:kworker.0 Request processed in 52us (328us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502494.802826:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 382 00000100:00100000:0.0:1777502494.802846:0:187553:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562074880 00000100:00100000:0.0:1777502494.802851:0:187553:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562074880 00000100:00080000:3.0:1777502494.802852:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:2.0:1777502494.802885:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4479b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562075136:0@lo:400:kworker.0 00000100:00100000:0.0:1777502494.802900:0:187553:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 263 00000100:00100000:2.0:1777502494.802905:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44798700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3312:1863837562075648:0@lo:400:kworker.0 00000100:00100000:3.0:1777502494.802910:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd63da6680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0000_UUID+5:3312:x1863837562075520:12345-0@lo:400:kworker.0 Request processed in 128us (241us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502494.802920:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 449 00000100:00100000:0.0:1777502494.802927:0:187553:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc547b1c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3311:x1863837562074880:12345-0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1777502494.802936:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3312:1863837562075520:0@lo:400:kworker.0 00002000:00000024:0.0:1777502494.802945:0:187553:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 42008576 pending 0 free 3831496704 avail 3829399552 00000100:00080000:3.0:1777502494.802948:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1777502494.802981:0:187553:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc547b1c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3311:x1863837562074880:12345-0@lo:13:osp-pre-0-0.0 Request processed in 72us (358us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502494.802986:0:187553:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 263 00000100:00100000:3.0:1777502494.803084:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4479a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3311:1863837562074880:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1777502494.803106:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4479b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3309:1863837562075264:0@lo:400:kworker.0 00000100:00100000:2.0:1777502494.803111:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4479aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:3310:1863837562075392:0@lo:400:kworker.0 00000100:00100000:0.0:1777502494.803137:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502494.803180:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3309:1863837562075008:0@lo:13:osp-pre-1-0.0 00000100:00100000:3.0:1777502494.803207:0:205381:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562075264 00000100:00100000:3.0:1777502494.803211:0:205381:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562075264 00000100:00100000:0.0:1777502494.803260:0:205375:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 471 00000100:00100000:0.0:1777502494.803271:0:205375:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc44799c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_007:lustre-MDT0000-mdtlov_UUID+5:3309:x1863837562075264:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502494.803325:0:205375:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc44799c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_007:lustre-MDT0000-mdtlov_UUID+5:3309:x1863837562075264:12345-0@lo:400:kworker.0 Request processed in 57us (188us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502494.803334:0:205375:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 471 00000100:00100000:2.0:1777502494.803353:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4479b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3309:1863837562075264:0@lo:400:kworker.0 00010000:00010000:2.0:1777502495.185691:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502496.209703:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502497.233775:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502498.257790:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502499.281699:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1777502499.485070:0:3304:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502499.485151:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567207424 00000100:00100000:2.0:1777502499.485157:0:3304:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502499.485163:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567207424 00000100:00100000:2.0:1777502499.485211:0:3304:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:0.0:1777502499.485226:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567207552 00000100:00100000:0.0:1777502499.485232:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567207552 00000100:00100000:3.0:1777502499.485273:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 383 00000100:00100000:2.0:1777502499.485280:0:3304:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502499.485289:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45ccc450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:c3865704-aa9b-444e-8420-b5af90b3c217+9:2370:x1863837567207424:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:0.0:1777502499.485302:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 450 00000100:00100000:0.0:1777502499.485314:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ccb2d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0775d2ab-558e-41f1-82c0-14244d133390+15:2371:x1863837567207552:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:2.0:1777502499.485335:0:205378:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567207808 00000100:00100000:2.0:1777502499.485341:0:205378:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567207808 00000100:00100000:3.0:1777502499.485370:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45ccc450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:c3865704-aa9b-444e-8420-b5af90b3c217+9:2370:x1863837567207424:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 86us (307us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502499.485374:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ccb2d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0775d2ab-558e-41f1-82c0-14244d133390+15:2371:x1863837567207552:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 60us (217us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502499.485379:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 383 00000100:00100000:0.0:1777502499.485389:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 450 00000100:00100000:2.0:1777502499.485403:0:205378:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567207680 00000100:00100000:2.0:1777502499.485407:0:205378:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567207680 00000100:00100000:0.0:1777502499.485453:0:189105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 472 00000100:00100000:2.0:1777502499.485463:0:205378:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 473 00000100:00100000:0.0:1777502499.485468:0:189105:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ccb2680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_005:0775d2ab-558e-41f1-82c0-14244d133390+7:2371:x1863837567207808:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:2.0:1777502499.485479:0:205378:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ccb1880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_010:0775d2ab-558e-41f1-82c0-14244d133390+6:2370:x1863837567207680:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:2.0:1777502499.485518:0:205378:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ccb1880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_010:0775d2ab-558e-41f1-82c0-14244d133390+6:2370:x1863837567207680:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 45us (241us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502499.485529:0:205378:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 473 00000100:00100000:0.0:1777502499.485531:0:189105:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ccb2680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_005:0775d2ab-558e-41f1-82c0-14244d133390+7:2371:x1863837567207808:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 64us (317us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502499.485541:0:189105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 472 00000100:00080000:2.0:1777502499.485558:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:2.0:1777502499.485565:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:2.0:1777502499.921787:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6f9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3309:1863837562075776:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1777502499.921807:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3310:1863837562075904:0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1777502499.921873:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502499.921957:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4479aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3309:1863837562076288:0@lo:400:kworker.0 00000100:00100000:2.0:1777502499.921988:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502499.921993:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4479b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562076160:0@lo:400:kworker.0 00000100:00100000:2.0:1777502499.922014:0:3309:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1777502499.922023:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4479a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3309:1863837562076416:0@lo:400:kworker.0 00000100:00100000:0.0:1777502499.922031:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502499.922057:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [2->3] 00000100:00100000:0.0:1777502499.922065:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44798a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562076032:0@lo:400:kworker.0 00000100:00100000:2.0:1777502499.922071:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502499.922101:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502499.922137:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3312:1863837562076544:0@lo:400:kworker.0 00000100:00100000:2.0:1777502499.922137:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562076288 00000100:00100000:2.0:1777502499.922146:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562076288 00000100:00100000:3.0:1777502499.922156:0:205383:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562076160 00000100:00100000:3.0:1777502499.922163:0:205383:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562076160 00000100:00100000:2.0:1777502499.922201:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562076416 00000100:00100000:0.0:1777502499.922203:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502499.922206:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562076416 00000100:00100000:3.0:1777502499.922227:0:205383:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 474 00000100:00100000:3.0:1777502499.922237:0:205383:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc44798700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_015:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562076160:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502499.922246:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562076544 00000100:00100000:0.0:1777502499.922247:0:187552:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562075776 00000100:00100000:2.0:1777502499.922250:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562076544 00000100:00100000:0.0:1777502499.922252:0:187552:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562075776 00000100:00100000:2.0:1777502499.922280:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 451 00000100:00100000:2.0:1777502499.922288:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6fb100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0000_UUID+5:3309:x1863837562076288:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502499.922298:0:187552:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 264 00000100:00100000:0.0:1777502499.922307:0:187552:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6f9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3309:x1863837562075776:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:0.0:1777502499.922332:0:187552:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 43712512 pending 0 free 3801088000 avail 3798990848 00000100:00100000:2.0:1777502499.922337:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6fb100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0000_UUID+5:3309:x1863837562076288:12345-0@lo:400:kworker.0 Request processed in 47us (346us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502499.922342:0:205383:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc44798700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_015:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562076160:12345-0@lo:400:kworker.0 Request processed in 106us (311us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502499.922352:0:205383:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 474 00000100:00100000:2.0:1777502499.922358:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 451 00000100:00100000:2.0:1777502499.922385:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 452 00000100:00100000:0.0:1777502499.922389:0:187552:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6f9c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3309:x1863837562075776:12345-0@lo:13:osp-pre-1-0.0 Request processed in 81us (516us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502499.922399:0:187552:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 264 00000100:00100000:2.0:1777502499.922399:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6fa680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3309:x1863837562076416:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502499.922438:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 453 00000100:00100000:2.0:1777502499.922441:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6fa680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3309:x1863837562076416:12345-0@lo:400:kworker.0 Request processed in 46us (373us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502499.922448:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 452 00000100:00100000:3.0:1777502499.922452:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4479b100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3312:x1863837562076544:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502499.922466:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502499.922479:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562076032 00000100:00100000:2.0:1777502499.922483:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562076032 00000100:00100000:2.0:1777502499.922515:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 384 00000100:00100000:3.0:1777502499.922515:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4479b100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3312:x1863837562076544:12345-0@lo:400:kworker.0 Request processed in 67us (312us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502499.922517:0:187553:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562075904 00000100:00100000:0.0:1777502499.922522:0:187553:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562075904 00000100:00100000:2.0:1777502499.922531:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3312:x1863837562076032:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502499.922537:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 453 00000100:00100000:0.0:1777502499.922554:0:187553:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 265 00000100:00100000:0.0:1777502499.922564:0:187553:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493f8700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3310:x1863837562075904:12345-0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1777502499.922567:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3312:x1863837562076032:12345-0@lo:400:kworker.0 Request processed in 43us (465us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502499.922572:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 384 00000100:00080000:3.0:1777502499.922597:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:3.0:1777502499.922603:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00002000:00000024:0.0:1777502499.922604:0:187553:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 42008576 pending 0 free 3831496704 avail 3829399552 00000100:00100000:0.0:1777502499.922652:0:187553:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493f8700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3310:x1863837562075904:12345-0@lo:13:osp-pre-0-0.0 Request processed in 89us (186us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502499.922661:0:187553:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 265 00000100:00100000:2.0:1777502499.922673:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4479a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:3309:1863837562076416:0@lo:400:kworker.0 00000100:00100000:2.0:1777502499.922695:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4479aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3309:1863837562076288:0@lo:400:kworker.0 00000100:00100000:0.0:1777502499.922725:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3310:1863837562075904:0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1777502499.922726:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6f9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3309:1863837562075776:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1777502499.922796:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44798a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562076032:0@lo:400:kworker.0 00000100:00100000:2.0:1777502499.922814:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3312:1863837562076544:0@lo:400:kworker.0 00000100:00100000:2.0:1777502499.922830:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4479b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562076160:0@lo:400:kworker.0 00010000:00010000:2.0:1777502500.305719:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502501.329737:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502502.354663:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502503.377745:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502504.401741:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1777502504.604948:0:3303:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502504.605010:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567208064 00000100:00100000:0.0:1777502504.605026:0:3303:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502504.605032:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567208064 00000100:00100000:0.0:1777502504.605071:0:3303:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502504.605081:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567207936 00000100:00100000:3.0:1777502504.605094:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567207936 00000100:00100000:2.0:1777502504.605102:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 454 00000100:00100000:0.0:1777502504.605116:0:3303:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502504.605122:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45702d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0775d2ab-558e-41f1-82c0-14244d133390+15:2371:x1863837567208064:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:3.0:1777502504.605155:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 385 00000100:00100000:3.0:1777502504.605166:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1bc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:c3865704-aa9b-444e-8420-b5af90b3c217+9:2370:x1863837567207936:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:0.0:1777502504.605173:0:188358:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567208192 00000100:00100000:0.0:1777502504.605179:0:188358:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567208192 00000100:00100000:2.0:1777502504.605197:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45702d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0775d2ab-558e-41f1-82c0-14244d133390+15:2371:x1863837567208064:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 80us (259us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502504.605205:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1bc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:c3865704-aa9b-444e-8420-b5af90b3c217+9:2370:x1863837567207936:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 40us (180us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502504.605210:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 454 00000100:00100000:3.0:1777502504.605227:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 385 00000100:00100000:0.0:1777502504.605231:0:188358:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567208320 00000100:00100000:0.0:1777502504.605241:0:188358:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567208320 00000100:00080000:3.0:1777502504.605263:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:3.0:1777502504.605269:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1777502504.605280:0:205380:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 475 00000100:00100000:0.0:1777502504.605289:0:188358:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 476 00000100:00100000:2.0:1777502504.605293:0:205380:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45700700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_012:0775d2ab-558e-41f1-82c0-14244d133390+6:2370:x1863837567208192:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:0.0:1777502504.605305:0:188358:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45701c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:0775d2ab-558e-41f1-82c0-14244d133390+7:2371:x1863837567208320:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:2.0:1777502504.605335:0:205380:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45700700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_012:0775d2ab-558e-41f1-82c0-14244d133390+6:2370:x1863837567208192:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 46us (263us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502504.605346:0:205380:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 475 00000100:00100000:0.0:1777502504.605347:0:188358:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45701c00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_003:0775d2ab-558e-41f1-82c0-14244d133390+7:2371:x1863837567208320:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 45us (231us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502504.605355:0:188358:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 476 00000100:00100000:2.0:1777502505.041779:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6f9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562076672:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1777502505.041833:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc547b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562076928:0@lo:400:kworker.0 00000100:00100000:2.0:1777502505.041869:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502505.041887:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502505.041952:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd509d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562077184:0@lo:400:kworker.0 00000100:00100000:0.0:1777502505.041964:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd509d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-OST0000_UUID:3310:1863837562077440:0@lo:400:kworker.0 00000100:00100000:2.0:1777502505.041980:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502505.041994:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc547b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3311:1863837562077056:0@lo:400:kworker.0 00000100:00100000:0.0:1777502505.042007:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502505.042033:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562076928 00000100:00100000:0.0:1777502505.042042:0:3310:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1777502505.042042:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562076928 00000100:00100000:3.0:1777502505.042043:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502505.042053:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c69880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3310:1863837562076800:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1777502505.042077:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd509d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3311:1863837562077568:0@lo:400:kworker.0 00000100:00100000:2.0:1777502505.042094:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 386 00000100:00100000:0.0:1777502505.042100:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502505.042107:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee19850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3310:x1863837562076928:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502505.042116:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502505.042129:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd509d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:3310:1863837562077312:0@lo:400:kworker.0 00000100:00100000:2.0:1777502505.042163:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee19850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3310:x1863837562076928:12345-0@lo:400:kworker.0 Request processed in 58us (276us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502505.042171:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 386 00000100:00100000:0.0:1777502505.042177:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502505.042190:0:205379:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562077184 00000100:00100000:3.0:1777502505.042196:0:205379:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562077184 00000100:00100000:2.0:1777502505.042213:0:205377:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562077056 00000100:00100000:2.0:1777502505.042217:0:205377:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562077056 00000100:00100000:0.0:1777502505.042219:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc547b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562076928:0@lo:400:kworker.0 00000100:00100000:3.0:1777502505.042257:0:205379:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 477 00000100:00100000:2.0:1777502505.042263:0:205377:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 478 00000100:00100000:3.0:1777502505.042269:0:205379:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6f9180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_011:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562077184:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502505.042274:0:205377:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d2680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_009:lustre-MDT0000-mdtlov_UUID+5:3311:x1863837562077056:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502505.042333:0:205379:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6f9180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_011:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562077184:12345-0@lo:400:kworker.0 Request processed in 66us (351us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502505.042338:0:205377:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d2680 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_009:lustre-MDT0000-mdtlov_UUID+5:3311:x1863837562077056:12345-0@lo:400:kworker.0 Request processed in 66us (296us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502505.042346:0:205379:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 477 00000100:00100000:2.0:1777502505.042354:0:205377:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 478 00000100:00100000:2.0:1777502505.042384:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562077568 00000100:00100000:3.0:1777502505.042384:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562077440 00000100:00100000:2.0:1777502505.042389:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562077568 00000100:00100000:3.0:1777502505.042391:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562077440 00000100:00100000:2.0:1777502505.042422:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562077312 00000100:00100000:2.0:1777502505.042425:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562077312 00000100:00100000:3.0:1777502505.042440:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 455 00000100:00100000:3.0:1777502505.042452:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d0a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3311:x1863837562077568:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502505.042459:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 456 00000100:00100000:2.0:1777502505.042469:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c6b100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0000_UUID+5:3310:x1863837562077440:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502505.042506:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d0a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-OST0001_UUID+5:3311:x1863837562077568:12345-0@lo:400:kworker.0 Request processed in 56us (390us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502505.042515:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 455 00000100:00100000:2.0:1777502505.042521:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c6b100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0000_UUID+5:3310:x1863837562077440:12345-0@lo:400:kworker.0 Request processed in 53us (514us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502505.042528:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 456 00000100:00100000:3.0:1777502505.042540:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 457 00000100:00100000:3.0:1777502505.042550:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c6aa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:3310:x1863837562077312:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502505.042561:0:189738:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562076672 00000100:00100000:2.0:1777502505.042565:0:189738:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562076672 00000100:00100000:2.0:1777502505.042611:0:189738:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562076800 00000100:00100000:2.0:1777502505.042613:0:189738:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562076800 00000100:00100000:3.0:1777502505.042618:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c6aa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:3310:x1863837562077312:12345-0@lo:400:kworker.0 Request processed in 68us (441us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502505.042637:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 457 00000100:00100000:2.0:1777502505.042640:0:189738:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 266 00000100:00100000:0.0:1777502505.042650:0:187552:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 267 00000100:00100000:2.0:1777502505.042654:0:189738:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6f9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562076672:12345-0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1777502505.042665:0:187552:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c6b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3310:x1863837562076800:12345-0@lo:13:osp-pre-0-0.0 00000100:00080000:3.0:1777502505.042670:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:3.0:1777502505.042676:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00002000:00000024:2.0:1777502505.042683:0:189738:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 43712512 pending 0 free 3801088000 avail 3798990848 00002000:00000024:0.0:1777502505.042689:0:187552:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 42008576 pending 0 free 3831496704 avail 3829399552 00000100:00100000:2.0:1777502505.042731:0:189738:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6f9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562076672:12345-0@lo:13:osp-pre-1-0.0 Request processed in 82us (864us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502505.042737:0:189738:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 266 00000100:00100000:3.0:1777502505.042744:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc547b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3311:1863837562077056:0@lo:400:kworker.0 00000100:00100000:3.0:1777502505.042786:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd509d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3311:1863837562077568:0@lo:400:kworker.0 00000100:00100000:0.0:1777502505.042787:0:187552:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c6b480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3310:x1863837562076800:12345-0@lo:13:osp-pre-0-0.0 Request processed in 124us (688us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502505.042800:0:187552:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 267 00000100:00100000:2.0:1777502505.042836:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c69880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3310:1863837562076800:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1777502505.042844:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd509d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562077184:0@lo:400:kworker.0 00000100:00100000:2.0:1777502505.042856:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd509d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:3310:1863837562077312:0@lo:400:kworker.0 00000100:00100000:3.0:1777502505.042872:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6f9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562076672:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1777502505.042876:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd509d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-OST0000_UUID:3310:1863837562077440:0@lo:400:kworker.0 00010000:00010000:0.0:1777502505.425792:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502506.450865:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502507.473722:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502508.497703:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502509.521765:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1777502509.724822:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502509.724888:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502509.724903:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567208576 00000100:00100000:2.0:1777502509.724914:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567208576 00000100:00100000:3.0:1777502509.724937:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:0.0:1777502509.724958:0:205382:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567208832 00000100:00100000:0.0:1777502509.724977:0:205382:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567208832 00000100:00100000:3.0:1777502509.724983:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502509.724995:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 458 00000100:00100000:2.0:1777502509.725010:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d1c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0775d2ab-558e-41f1-82c0-14244d133390+15:2372:x1863837567208576:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:3.0:1777502509.725021:0:187548:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567208704 00000100:00100000:3.0:1777502509.725025:0:187548:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567208704 00000100:00100000:0.0:1777502509.725051:0:205382:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 479 00000100:00100000:0.0:1777502509.725063:0:205382:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_014:0775d2ab-558e-41f1-82c0-14244d133390+7:2372:x1863837567208832:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:3.0:1777502509.725078:0:187548:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 480 00000100:00100000:3.0:1777502509.725090:0:187548:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d2300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:0775d2ab-558e-41f1-82c0-14244d133390+6:2373:x1863837567208704:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:2.0:1777502509.725100:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d1c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0775d2ab-558e-41f1-82c0-14244d133390+15:2372:x1863837567208576:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 93us (286us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502509.725110:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 458 00000100:00100000:0.0:1777502509.725115:0:205382:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d0700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_014:0775d2ab-558e-41f1-82c0-14244d133390+7:2372:x1863837567208832:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 52us (225us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502509.725125:0:205382:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 479 00000100:00100000:3.0:1777502509.725142:0:187548:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d2300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:0775d2ab-558e-41f1-82c0-14244d133390+6:2373:x1863837567208704:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 54us (160us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502509.725149:0:187548:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 480 00000100:00100000:2.0:1777502509.725163:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567208448 00000100:00100000:2.0:1777502509.725168:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567208448 00000100:00080000:3.0:1777502509.725174:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00080000:3.0:1777502509.725179:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:2.0:1777502509.725206:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 387 00000100:00100000:2.0:1777502509.725216:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1a050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:c3865704-aa9b-444e-8420-b5af90b3c217+9:2373:x1863837567208448:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:2.0:1777502509.725246:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1a050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:c3865704-aa9b-444e-8420-b5af90b3c217+9:2373:x1863837567208448:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 32us (312us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502509.725254:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 387 00000100:00100000:0.0:1777502510.161830:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:0.0:1777502510.161864:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3311:1863837562077696:0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1777502510.161890:0:3310:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1777502510.161905:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6fb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3310:1863837562077824:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1777502510.161940:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502510.162140:0:189738:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562077696 00000100:00100000:2.0:1777502510.162148:0:189738:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562077696 00000100:00100000:2.0:1777502510.162213:0:189738:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 268 00000100:00100000:2.0:1777502510.162225:0:189738:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c68000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3311:x1863837562077696:12345-0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1777502510.162233:0:3309:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1777502510.162247:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c69500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562077952:0@lo:400:kworker.0 00002000:00000024:2.0:1777502510.162262:0:189738:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 42008576 pending 0 free 3831496704 avail 3829399552 00000100:00100000:3.0:1777502510.162291:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502510.162316:0:189738:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c68000 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3311:x1863837562077696:12345-0@lo:13:osp-pre-0-0.0 Request processed in 93us (377us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502510.162326:0:189738:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 268 00000100:00100000:3.0:1777502510.162366:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562077952 00000100:00100000:2.0:1777502510.162371:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502510.162410:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562077952 00000100:00100000:2.0:1777502510.162419:0:205168:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562077824 00000100:00100000:2.0:1777502510.162425:0:205168:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562077824 00000100:00080000:3.0:1777502510.162459:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:2.0:1777502510.162465:0:205168:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 269 00000100:00080000:3.0:1777502510.162466:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1777502510.162477:0:205168:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6f9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3310:x1863837562077824:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:2.0:1777502510.162501:0:205168:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 43712512 pending 0 free 3801088000 avail 3798990848 00000100:00100000:3.0:1777502510.162522:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 388 00000100:00100000:2.0:1777502510.162542:0:205168:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6f9f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3310:x1863837562077824:12345-0@lo:13:osp-pre-1-0.0 Request processed in 64us (170us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502510.162542:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee19050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3309:x1863837562077952:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502510.162555:0:205168:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 269 00000100:00100000:2.0:1777502510.162613:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc547b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-OST0001_UUID:3310:1863837562078464:0@lo:400:kworker.0 00000100:00100000:3.0:1777502510.162615:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee19050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3309:x1863837562077952:12345-0@lo:400:kworker.0 Request processed in 77us (323us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502510.162626:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 388 00000100:00100000:2.0:1777502510.162652:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502510.162706:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6fb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3310:1863837562077824:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1777502510.162728:0:3310:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1777502510.162736:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c69880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-OST0000_UUID:3310:1863837562078336:0@lo:400:kworker.0 00000100:00100000:2.0:1777502510.162765:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502510.162806:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562078464 00000100:00100000:2.0:1777502510.162811:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562078464 00000100:00100000:2.0:1777502510.162857:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562078336 00000100:00100000:2.0:1777502510.162862:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562078336 00000100:00100000:2.0:1777502510.162891:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 459 00000100:00100000:2.0:1777502510.162900:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6f8a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0001_UUID+5:3310:x1863837562078464:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502510.162944:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6f8a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0001_UUID+5:3310:x1863837562078464:12345-0@lo:400:kworker.0 Request processed in 45us (293us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502510.162951:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 459 00000100:00100000:2.0:1777502510.162975:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 460 00000100:00100000:2.0:1777502510.162985:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6fb800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0000_UUID+5:3310:x1863837562078336:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502510.162992:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0000_UUID:3312:1863837562078208:0@lo:400:kworker.0 00000100:00100000:2.0:1777502510.163030:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6fb800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0000_UUID+5:3310:x1863837562078336:12345-0@lo:400:kworker.0 Request processed in 46us (263us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502510.163046:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 460 00000100:00100000:0.0:1777502510.163085:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502510.163094:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c69880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-OST0000_UUID:3310:1863837562078336:0@lo:400:kworker.0 00000100:00100000:2.0:1777502510.163115:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc547b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-OST0001_UUID:3310:1863837562078464:0@lo:400:kworker.0 00000100:00100000:0.0:1777502510.163121:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:0.0:1777502510.163131:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562078080:0@lo:400:kworker.0 00000100:00100000:0.0:1777502510.163162:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502510.163202:0:205375:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562078080 00000100:00100000:0.0:1777502510.163206:0:205375:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562078080 00000100:00100000:3.0:1777502510.163207:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562078208 00000100:00100000:3.0:1777502510.163213:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562078208 00000100:00100000:0.0:1777502510.163248:0:205375:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 481 00000100:00100000:2.0:1777502510.163250:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c69500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562077952:0@lo:400:kworker.0 00000100:00100000:0.0:1777502510.163261:0:205375:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc547b3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_007:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562078080:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502510.163308:0:205375:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc547b3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_007:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562078080:12345-0@lo:400:kworker.0 Request processed in 52us (146us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502510.163316:0:205375:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 481 00000100:00100000:2.0:1777502510.163325:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 461 00000100:00100000:2.0:1777502510.163337:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc547b2d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:3312:x1863837562078208:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502510.163368:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562078080:0@lo:400:kworker.0 00000100:00100000:0.0:1777502510.163392:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-MDT0000_UUID:3312:1863837562078208:0@lo:400:kworker.0 00000100:00100000:2.0:1777502510.163392:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc547b2d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:lustre-MDT0000-lwp-MDT0000_UUID+5:3312:x1863837562078208:12345-0@lo:400:kworker.0 Request processed in 58us (356us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502510.163407:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 461 00000100:00100000:0.0:1777502510.163455:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3311:1863837562077696:0@lo:13:osp-pre-0-0.0 00010000:00010000:3.0:1777502510.545776:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502511.571979:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502512.593710:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502513.617767:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502514.641776:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1777502514.844935:0:3304:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502514.844982:0:3304:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502514.845001:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567208960 00000100:00100000:2.0:1777502514.845011:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567208960 00000100:00100000:3.0:1777502514.845024:0:3304:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:0.0:1777502514.845034:0:189105:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567209216 00000100:00100000:0.0:1777502514.845040:0:189105:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567209216 00000100:00100000:2.0:1777502514.845073:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 389 00000100:00100000:3.0:1777502514.845080:0:3304:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502514.845084:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1a450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:c3865704-aa9b-444e-8420-b5af90b3c217+9:2370:x1863837567208960:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:0.0:1777502514.845099:0:189105:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567209344 00000100:00100000:0.0:1777502514.845104:0:189105:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567209344 00000100:00100000:3.0:1777502514.845127:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567209088 00000100:00100000:3.0:1777502514.845132:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567209088 00000100:00100000:0.0:1777502514.845145:0:189105:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 482 00000100:00100000:0.0:1777502514.845158:0:189105:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_005:0775d2ab-558e-41f1-82c0-14244d133390+6:2370:x1863837567209216:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:2.0:1777502514.845166:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1a450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:c3865704-aa9b-444e-8420-b5af90b3c217+9:2370:x1863837567208960:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 83us (377us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502514.845172:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 462 00000100:00100000:2.0:1777502514.845174:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 389 00000100:00100000:3.0:1777502514.845182:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d2d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0775d2ab-558e-41f1-82c0-14244d133390+15:2370:x1863837567209088:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:0.0:1777502514.845202:0:189105:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d3100 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_005:0775d2ab-558e-41f1-82c0-14244d133390+6:2370:x1863837567209216:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 47us (219us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502514.845211:0:189105:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 482 00000100:00100000:3.0:1777502514.845220:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d2d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0775d2ab-558e-41f1-82c0-14244d133390+15:2370:x1863837567209088:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 39us (195us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502514.845226:0:205368:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 483 00000100:00100000:3.0:1777502514.845228:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 462 00000100:00100000:2.0:1777502514.845235:0:205368:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d1500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_006:0775d2ab-558e-41f1-82c0-14244d133390+7:2370:x1863837567209344:12345-192.168.202.27@tcp:400:kworker.0 00000100:00080000:0.0:1777502514.845242:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:0.0:1777502514.845248:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1777502514.845262:0:205368:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d1500 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_006:0775d2ab-558e-41f1-82c0-14244d133390+7:2370:x1863837567209344:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 29us (184us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502514.845269:0:205368:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 483 00000100:00100000:0.0:1777502515.281762:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f595c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562078592:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1777502515.281845:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502515.281918:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:0.0:1777502515.281927:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6fb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562078720:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1777502515.281962:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502515.282121:0:189738:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562078592 00000100:00100000:3.0:1777502515.282130:0:189738:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562078592 00000100:00100000:3.0:1777502515.282192:0:189738:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562078720 00000100:00100000:3.0:1777502515.282196:0:189738:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562078720 00000100:00100000:2.0:1777502515.282218:0:187553:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 270 00000100:00100000:3.0:1777502515.282228:0:189738:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 271 00000100:00100000:2.0:1777502515.282230:0:187553:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f596d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562078592:12345-0@lo:13:osp-pre-0-0.0 00000100:00080000:0.0:1777502515.282234:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:0.0:1777502515.282240:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1777502515.282249:0:189738:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f595f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562078720:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:3.0:1777502515.282378:0:189738:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 43712512 pending 0 free 3801088000 avail 3798990848 00002000:00000024:2.0:1777502515.282407:0:187553:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 42008576 pending 0 free 3831496704 avail 3829399552 00000100:00100000:3.0:1777502515.282432:0:189738:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f595f80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562078720:12345-0@lo:13:osp-pre-1-0.0 Request processed in 183us (469us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502515.282434:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f595180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562078848:0@lo:400:kworker.0 00000100:00100000:3.0:1777502515.282446:0:189738:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 271 00000100:00100000:2.0:1777502515.282447:0:187553:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f596d80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562078592:12345-0@lo:13:osp-pre-0-0.0 Request processed in 217us (602us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502515.282454:0:187553:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 270 00000100:00100000:0.0:1777502515.282473:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502515.282514:0:3309:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1777502515.282520:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f594000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3309:1863837562078976:0@lo:400:kworker.0 00000100:00100000:3.0:1777502515.282551:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562078848 00000100:00100000:0.0:1777502515.282552:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502515.282558:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562078848 00000100:00100000:0.0:1777502515.282611:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f595880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3311:1863837562079104:0@lo:400:kworker.0 00000100:00100000:2.0:1777502515.282631:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562078976 00000100:00100000:2.0:1777502515.282636:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562078976 00000100:00100000:0.0:1777502515.282639:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502515.282665:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1777502515.282669:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 390 00000100:00100000:0.0:1777502515.282674:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f596300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3311:1863837562079232:0@lo:400:kworker.0 00000100:00100000:2.0:1777502515.282674:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562079104 00000100:00100000:2.0:1777502515.282677:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562079104 00000100:00100000:3.0:1777502515.282680:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee18450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3309:x1863837562078848:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502515.282705:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 463 00000100:00100000:2.0:1777502515.282716:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f594700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0000_UUID+5:3309:x1863837562078976:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502515.282718:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502515.282736:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee18450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3309:x1863837562078848:12345-0@lo:400:kworker.0 Request processed in 59us (262us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502515.282745:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 390 00000100:00100000:0.0:1777502515.282760:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562079232 00000100:00100000:2.0:1777502515.282762:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f594700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-MDT0000_UUID+5:3309:x1863837562078976:12345-0@lo:400:kworker.0 Request processed in 50us (211us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502515.282764:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562079232 00000100:00100000:2.0:1777502515.282772:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 463 00000100:00100000:3.0:1777502515.282788:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 464 00000100:00100000:2.0:1777502515.282810:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 465 00000100:00100000:3.0:1777502515.282813:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f597100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3311:x1863837562079104:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502515.282819:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f596680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0001_UUID+5:3311:x1863837562079232:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502515.282856:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6fb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562078720:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1777502515.282862:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f596680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0001_UUID+5:3311:x1863837562079232:12345-0@lo:400:kworker.0 Request processed in 42us (145us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502515.282862:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f597100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3311:x1863837562079104:12345-0@lo:400:kworker.0 Request processed in 63us (222us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502515.282870:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 465 00000100:00100000:3.0:1777502515.282877:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 464 00000100:00100000:0.0:1777502515.282878:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f595c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562078592:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1777502515.282924:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f596300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3311:1863837562079232:0@lo:400:kworker.0 00000100:00100000:0.0:1777502515.282933:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f594000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3309:1863837562078976:0@lo:400:kworker.0 00000100:00100000:3.0:1777502515.282944:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f595880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3311:1863837562079104:0@lo:400:kworker.0 00000100:00100000:0.0:1777502515.282949:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f595180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562078848:0@lo:400:kworker.0 00010000:00010000:2.0:1777502515.665696:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502516.689753:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502517.713722:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502518.737720:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502519.761939:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1777502519.964829:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502519.964887:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502519.964894:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567209472 00000100:00100000:2.0:1777502519.964904:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567209472 00000100:00100000:3.0:1777502519.964934:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:0.0:1777502519.964964:0:187550:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567209728 00000100:00100000:2.0:1777502519.964964:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 391 00000100:00100000:0.0:1777502519.964971:0:187550:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567209728 00000100:00100000:3.0:1777502519.964971:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502519.964976:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee18850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:c3865704-aa9b-444e-8420-b5af90b3c217+9:2370:x1863837567209472:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:3.0:1777502519.965029:0:205386:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567209856 00000100:00100000:3.0:1777502519.965033:0:205386:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567209856 00000100:00100000:0.0:1777502519.965038:0:187550:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 484 00000100:00100000:2.0:1777502519.965038:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee18850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:c3865704-aa9b-444e-8420-b5af90b3c217+9:2370:x1863837567209472:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 63us (224us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502519.965052:0:187550:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:0775d2ab-558e-41f1-82c0-14244d133390+6:2370:x1863837567209728:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:2.0:1777502519.965053:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 391 00000100:00100000:3.0:1777502519.965074:0:205386:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 485 00000100:00100000:3.0:1777502519.965083:0:205386:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d1180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_018:0775d2ab-558e-41f1-82c0-14244d133390+7:2370:x1863837567209856:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:0.0:1777502519.965103:0:187550:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d3480 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_001:0775d2ab-558e-41f1-82c0-14244d133390+6:2370:x1863837567209728:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 55us (217us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502519.965115:0:205386:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d1180 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_018:0775d2ab-558e-41f1-82c0-14244d133390+7:2370:x1863837567209856:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 32us (142us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502519.965116:0:187550:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 484 00000100:00100000:3.0:1777502519.965126:0:205386:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 485 00000100:00100000:0.0:1777502519.965153:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567209600 00000100:00100000:0.0:1777502519.965157:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567209600 00000100:00080000:3.0:1777502519.965176:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:3.0:1777502519.965181:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:0.0:1777502519.965200:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 466 00000100:00100000:0.0:1777502519.965210:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d1880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0775d2ab-558e-41f1-82c0-14244d133390+15:2370:x1863837567209600:12345-192.168.202.27@tcp:400:kworker.0 00000100:00100000:0.0:1777502519.965256:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d1880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0775d2ab-558e-41f1-82c0-14244d133390+15:2370:x1863837567209600:12345-192.168.202.27@tcp:400:kworker.0 Request processed in 48us (324us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502519.965263:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 466 00000100:00100000:0.0:1777502520.401925:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f595180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3309:1863837562079360:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1777502520.402000:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502520.402054:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f594000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3310:1863837562079488:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1777502520.402063:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f596a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3309:1863837562079872:0@lo:400:kworker.0 00000100:00100000:2.0:1777502520.402092:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502520.402113:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502520.402122:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f594e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:3310:1863837562080000:0@lo:400:kworker.0 00000100:00100000:2.0:1777502520.402159:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502520.402167:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f595c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562079616:0@lo:400:kworker.0 00000100:00100000:0.0:1777502520.402207:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502520.402260:0:187552:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562079360 00000100:00100000:2.0:1777502520.402267:0:187552:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562079360 00000100:00100000:0.0:1777502520.402270:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f597b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562079744:0@lo:400:kworker.0 00000100:00100000:0.0:1777502520.402300:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502520.402331:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f597800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3312:1863837562080256:0@lo:400:kworker.0 00000100:00080000:3.0:1777502520.402338:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1777502520.402355:0:205379:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562079872 00000100:00100000:3.0:1777502520.402359:0:205379:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562079872 00000100:00100000:0.0:1777502520.402364:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502520.402405:0:205379:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 486 00000100:00100000:3.0:1777502520.402417:0:205379:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f594a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_011:lustre-MDT0000-mdtlov_UUID+5:3309:x1863837562079872:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502520.402490:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562080000 00000100:00100000:2.0:1777502520.402495:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562080000 00000100:00100000:3.0:1777502520.402500:0:205379:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f594a80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_011:lustre-MDT0000-mdtlov_UUID+5:3309:x1863837562079872:12345-0@lo:400:kworker.0 Request processed in 85us (388us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502520.402509:0:205379:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 486 00000100:00100000:3.0:1777502520.402534:0:205379:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562079744 00000100:00100000:3.0:1777502520.402539:0:205379:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562079744 00000100:00100000:2.0:1777502520.402602:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562080256 00000100:00100000:3.0:1777502520.402602:0:205379:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 487 00000100:00100000:2.0:1777502520.402608:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562080256 00000100:00100000:3.0:1777502520.402613:0:205379:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f595880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_011:lustre-MDT0000-mdtlov_UUID+6:3312:x1863837562079744:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502520.402647:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 467 00000100:00100000:2.0:1777502520.402658:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6f8000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0000_UUID+5:3310:x1863837562080000:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502520.402665:0:205379:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f595880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_011:lustre-MDT0000-mdtlov_UUID+6:3312:x1863837562079744:12345-0@lo:400:kworker.0 Request processed in 53us (363us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502520.402674:0:205379:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 487 00000100:00100000:3.0:1777502520.402707:0:189738:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562079488 00000100:00100000:3.0:1777502520.402712:0:189738:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562079488 00000100:00100000:2.0:1777502520.402716:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6f8000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0000_UUID+5:3310:x1863837562080000:12345-0@lo:400:kworker.0 Request processed in 59us (556us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502520.402724:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 467 00000100:00100000:3.0:1777502520.402759:0:189738:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 272 00000100:00100000:2.0:1777502520.402764:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 468 00000100:00100000:3.0:1777502520.402773:0:189738:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f594380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3309:x1863837562079360:12345-0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1777502520.402777:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f596300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0001_UUID+5:3312:x1863837562080256:12345-0@lo:400:kworker.0 00002000:00000024:3.0:1777502520.402813:0:189738:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 42008576 pending 0 free 3831496704 avail 3829399552 00000100:00100000:2.0:1777502520.402821:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f596300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0001_UUID+5:3312:x1863837562080256:12345-0@lo:400:kworker.0 Request processed in 48us (457us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502520.402828:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 468 00000100:00100000:3.0:1777502520.402859:0:189738:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f594380 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3309:x1863837562079360:12345-0@lo:13:osp-pre-0-0.0 Request processed in 89us (860us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502520.402868:0:189738:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 272 00000100:00100000:2.0:1777502520.402892:0:187552:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 273 00000100:00100000:2.0:1777502520.402903:0:187552:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6fa300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3310:x1863837562079488:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:2.0:1777502520.402924:0:187552:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 43712512 pending 0 free 3801088000 avail 3798990848 00000100:00080000:3.0:1777502520.402954:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:2.0:1777502520.402964:0:187552:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6fa300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3310:x1863837562079488:12345-0@lo:13:osp-pre-1-0.0 Request processed in 64us (873us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502520.402976:0:187552:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 273 00000100:00100000:3.0:1777502520.402997:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f595180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3309:1863837562079360:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1777502520.403020:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f596a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3309:1863837562079872:0@lo:400:kworker.0 00000100:00100000:2.0:1777502520.403026:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f594000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3310:1863837562079488:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1777502520.403049:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f594e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-MDT0000_UUID:3310:1863837562080000:0@lo:400:kworker.0 00000100:00100000:2.0:1777502520.403111:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562079616 00000100:00100000:2.0:1777502520.403115:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562079616 00000100:00100000:2.0:1777502520.403156:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 392 00000100:00100000:2.0:1777502520.403166:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1c850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3311:x1863837562079616:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502520.403205:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1c850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3311:x1863837562079616:12345-0@lo:400:kworker.0 Request processed in 42us (999us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502520.403212:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 392 00000100:00100000:0.0:1777502520.403240:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f597b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562079744:0@lo:400:kworker.0 00000100:00100000:0.0:1777502520.403261:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f597800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3312:1863837562080256:0@lo:400:kworker.0 00000100:00100000:3.0:1777502520.403290:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f595c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562079616:0@lo:400:kworker.0 00000100:00100000:3.0:1777502520.403302:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f597480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3311:1863837562080128:0@lo:400:kworker.0 00000100:00100000:3.0:1777502520.403337:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502520.403383:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562080128 00000100:00100000:2.0:1777502520.403389:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562080128 00000100:00100000:2.0:1777502520.403426:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 469 00000100:00100000:2.0:1777502520.403435:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d3b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0000_UUID+5:3311:x1863837562080128:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502520.403482:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d3b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:lustre-MDT0000-lwp-OST0000_UUID+5:3311:x1863837562080128:12345-0@lo:400:kworker.0 Request processed in 47us (146us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502520.403490:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 469 00000100:00100000:2.0:1777502520.403542:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f597480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:3311:1863837562080128:0@lo:400:kworker.0 00010000:00010000:2.0:1777502520.785670:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502521.810704:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1777502521.944345:0:221224:0:(debug.c:645:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa07549afe000.ost_server_uuid 50 00000001:02000400:3.0:1777502522.510484:0:221272:0:(debug.c:645:libcfs_debug_mark_buffer()) DEBUG MARKER: osc.lustre-OST0000-osc-ffffa07549afe000.ost_server_uuid in FULL state after 0 sec 00010000:00010000:3.0:1777502522.834698:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1777502523.093037:0:221320:0:(debug.c:645:libcfs_debug_mark_buffer()) DEBUG MARKER: cleanup: ====================================================== 00000001:02000400:2.0:1777502523.676640:0:221368:0:(debug.c:645:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanityn test complete, duration 8354 sec ============== 18:42:02 (1777502522) 00010000:00010000:3.0:1777502523.859821:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1777502524.260700:0:221416:0:(debug.c:645:libcfs_debug_mark_buffer()) DEBUG MARKER: === sanityn: start cleanup 18:42:03 (1777502523) === 00000100:00100000:3.0:1777502524.531736:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502524.531830:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502524.531865:0:205388:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567210112 00000100:00100000:3.0:1777502524.531872:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502524.531887:0:205388:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567210112 00000100:00100000:0.0:1777502524.531893:0:186922:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567209984 00000100:00100000:0.0:1777502524.531898:0:186922:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567209984 00000100:00100000:3.0:1777502524.531929:0:191872:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567210240 00000100:00100000:3.0:1777502524.531933:0:191872:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567210240 00000100:00100000:2.0:1777502524.531958:0:205388:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 297 00000100:00100000:0.0:1777502524.531969:0:186922:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 298 00000100:00100000:2.0:1777502524.531974:0:205388:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd61bff450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_005:c3865704-aa9b-444e-8420-b5af90b3c217+11:2370:x1863837567210112:12345-192.168.202.27@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1777502524.531983:0:186922:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_001:c3865704-aa9b-444e-8420-b5af90b3c217+11:2373:x1863837567209984:12345-192.168.202.27@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1777502524.531988:0:191872:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 299 00010000:00010000:2.0:1777502524.531991:0:205388:0:(ldlm_lockd.c:1747:ldlm_request_cancel()) ### server-side cancel handler START: 1 locks, starting at 0 00010000:00010000:0.0:1777502524.531997:0:186922:0:(ldlm_lockd.c:1747:ldlm_request_cancel()) ### server-side cancel handler START: 1 locks, starting at 0 00000100:00100000:3.0:1777502524.532002:0:191872:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_004:c3865704-aa9b-444e-8420-b5af90b3c217+11:2370:x1863837567210240:12345-192.168.202.27@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1777502524.532012:0:191872:0:(ldlm_lockd.c:1747:ldlm_request_cancel()) ### server-side cancel handler START: 1 locks, starting at 0 00010000:00010000:2.0:1777502524.532013:0:205388:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGS lock: ffff8dfc4e51e800/0x4ad31463d9b9e86f lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x44801000000000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf82a954 expref: 11 pid: 186929 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502524.532014:0:186922:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGS lock: ffff8dfd80fd7400/0x4ad31463d9b9e868 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 11 type: PLN flags: 0x44801000000000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf82a92a expref: 11 pid: 186931 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502524.532026:0:191872:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGS lock: ffff8dfd80fd7000/0x4ad31463d9b9ea60 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 6 type: PLN flags: 0x44801000000000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf82aa57 expref: 11 pid: 186930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502524.532043:0:205388:0:(ldlm_lockd.c:1808:ldlm_request_cancel()) ### server-side cancel handler END 00010000:00010000:0.0:1777502524.532045:0:186922:0:(ldlm_lockd.c:1808:ldlm_request_cancel()) ### server-side cancel handler END 00010000:00010000:3.0:1777502524.532056:0:191872:0:(ldlm_lockd.c:1808:ldlm_request_cancel()) ### server-side cancel handler END 00000100:00100000:0.0:1777502524.532093:0:186922:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d3800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_001:c3865704-aa9b-444e-8420-b5af90b3c217+8:2373:x1863837567209984:12345-192.168.202.27@tcp:103:ldlm_bl.0 Request processed in 115us (264us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502524.532096:0:205388:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd61bff450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_005:c3865704-aa9b-444e-8420-b5af90b3c217+8:2370:x1863837567210112:12345-192.168.202.27@tcp:103:ldlm_bl.0 Request processed in 128us (367us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502524.532099:0:191872:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d1f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_004:c3865704-aa9b-444e-8420-b5af90b3c217+8:2370:x1863837567210240:12345-192.168.202.27@tcp:103:ldlm_bl.0 Request processed in 99us (226us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502524.532105:0:186922:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 298 00000100:00100000:2.0:1777502524.532115:0:205388:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 297 00000100:00100000:3.0:1777502524.532136:0:191872:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 299 00000100:00100000:2.0:1777502524.539032:0:3304:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:0.0:1777502524.539085:0:205385:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567210624 00000100:00100000:0.0:1777502524.539092:0:205385:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567210624 00000100:00080000:3.0:1777502524.539147:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00100000:0.0:1777502524.539164:0:205385:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 6848 00000100:00100000:0.0:1777502524.539176:0:205385:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6f8700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_017:0775d2ab-558e-41f1-82c0-14244d133390+7:2369:x1863837567210624:12345-192.168.202.27@tcp:9:umount.0 00000020:00000080:0.0:1777502524.539216:0:205385:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x4ad31463d9c2d15e 00000020:00010000:0.0:1777502524.539226:0:205385:0:(genops.c:1942:obd_stale_export_put()) Put export ffff8dfd4fd53000: total 0 00000100:00080000:0.0:1777502524.539232:0:205385:0:(import.c:65:import_set_state_nolock()) ffff8dfd4fd57000 : changing import state from FULL to CLOSED 00000001:00000004:0.0:1777502524.539270:0:205385:0:(tgt_lastrcvd.c:732:tgt_server_data_update()) lustre-OST0001_UUID: mount_count is 2, last_transno is 8589937082 00000020:00010000:2.0:1777502524.539271:0:188362:0:(genops.c:1928:obd_stale_export_get()) Get export ffff8dfd4fd53000: total 0 00010000:00010000:2.0:1777502524.539317:0:188362:0:(ldlm_lock.c:2652:ldlm_export_cancel_blocked_locks()) Export ffff8dfd4fd53000, canceled 0 locks, left on hash table 2. 00010000:00010000:2.0:1777502524.539388:0:188362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0001_UUID lock: 00000000901ee638/0x4ad31463d9c2e474 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x1569:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x44801000020000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf845596 expref: 6 pid: 188366 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502524.539445:0:188362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0001_UUID lock: 00000000d5786e0d/0x4ad31463d9c2e291 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x1568:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x44801000020000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf8454e0 expref: 5 pid: 187551 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502524.539476:0:188362:0:(ldlm_lock.c:2680:ldlm_export_cancel_locks()) Export ffff8dfd4fd53000, canceled 2 locks, left on hash table 0. 00010000:00010000:2.0:1777502524.539495:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502524.539499:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502524.539502:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1777502524.539640:0:205385:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6f8700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_017:0775d2ab-558e-41f1-82c0-14244d133390+2:2369:x1863837567210624:12345-192.168.202.27@tcp:9:umount.0 Request processed in 467us (613us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502524.539651:0:205385:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 6848 00000100:00100000:0.0:1777502524.541528:0:3303:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502524.541602:0:187548:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567210880 00000100:00100000:3.0:1777502524.541608:0:187548:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567210880 00000100:00080000:0.0:1777502524.541650:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:3.0:1777502524.541662:0:187548:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 6849 00000100:00100000:3.0:1777502524.541671:0:187548:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f597800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:0775d2ab-558e-41f1-82c0-14244d133390+6:2369:x1863837567210880:12345-192.168.202.27@tcp:9:umount.0 00000020:00000080:3.0:1777502524.541687:0:187548:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x4ad31463d9c2d157 00000020:00010000:3.0:1777502524.541692:0:187548:0:(genops.c:1942:obd_stale_export_put()) Put export ffff8dfd4fd51800: total 0 00000100:00080000:3.0:1777502524.541696:0:187548:0:(import.c:65:import_set_state_nolock()) ffff8dfd4fd50000 : changing import state from FULL to CLOSED 00000020:00010000:2.0:1777502524.541728:0:186924:0:(genops.c:1928:obd_stale_export_get()) Get export ffff8dfd4fd51800: total 0 00000001:00000004:3.0:1777502524.541732:0:187548:0:(tgt_lastrcvd.c:732:tgt_server_data_update()) lustre-OST0000_UUID: mount_count is 2, last_transno is 8589937482 00010000:00010000:2.0:1777502524.541733:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502524.541782:0:186924:0:(ldlm_lock.c:2652:ldlm_export_cancel_blocked_locks()) Export ffff8dfd4fd51800, canceled 0 locks, left on hash table 1. 00010000:00010000:2.0:1777502524.541850:0:186924:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: filter-lustre-OST0000_UUID lock: 00000000042e9186/0x4ad31463d9c2e466 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x1590:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4096->1048575) gid 0 flags: 0x44801000020000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf845588 expref: 5 pid: 205375 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502524.541892:0:186924:0:(ldlm_lock.c:2680:ldlm_export_cancel_locks()) Export ffff8dfd4fd51800, canceled 1 locks, left on hash table 0. 00010000:00010000:2.0:1777502524.541912:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502524.541915:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502524.541919:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1777502524.542018:0:187548:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f597800 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_000:0775d2ab-558e-41f1-82c0-14244d133390+2:2369:x1863837567210880:12345-192.168.202.27@tcp:9:umount.0 Request processed in 347us (495us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502524.542027:0:187548:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 6849 00000100:00100000:3.0:1777502524.545286:0:3302:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:2.0:1777502524.545327:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567212160 00000100:00100000:2.0:1777502524.545334:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567212160 00000100:00100000:2.0:1777502524.545383:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 16658 00000100:00100000:2.0:1777502524.545393:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d0000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0775d2ab-558e-41f1-82c0-14244d133390+15:2369:x1863837567212160:12345-192.168.202.27@tcp:39:umount.0 00000020:00000080:2.0:1777502524.545438:0:187571:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x4ad31463d9b9e87d 00000020:00010000:2.0:1777502524.545470:0:187571:0:(genops.c:1942:obd_stale_export_put()) Put export ffff8dfd66b1b800: total 0 00000100:00080000:2.0:1777502524.545474:0:187571:0:(import.c:65:import_set_state_nolock()) ffff8dfd66b1a800 : changing import state from FULL to CLOSED 00000001:00000004:2.0:1777502524.545493:0:187571:0:(tgt_lastrcvd.c:732:tgt_server_data_update()) lustre-MDT0000_UUID: mount_count is 2, last_transno is 8589949328 00000020:00010000:3.0:1777502524.545516:0:205387:0:(genops.c:1928:obd_stale_export_get()) Get export ffff8dfd66b1b800: total 0 00010000:00010000:3.0:1777502524.545557:0:205387:0:(ldlm_lock.c:2652:ldlm_export_cancel_blocked_locks()) Export ffff8dfd66b1b800, canceled 0 locks, left on hash table 9. 00010000:00010000:3.0:1777502524.545619:0:205387:0:(ldlm_inodebits.c:74:ldlm_reprocess_inodebits_queue()) --- Reprocess resource [0x200000bd1:0xc29:0x0].0x0 (ffff8dfc51877700) 00010000:00010000:3.0:1777502524.545623:0:205387:0:(ldlm_inodebits.c:76:ldlm_reprocess_inodebits_queue()) Hint 1a 00010000:00010000:3.0:1777502524.545629:0:205387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: mdt-lustre-MDT0000_UUID lock: ffff8dfc85598c00/0x4ad31463d9c2e283 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0xc29:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x44a01000000000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf8454d9 expref: 14 pid: 186937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502524.545654:0:205387:0:(ldlm_inodebits.c:74:ldlm_reprocess_inodebits_queue()) --- Reprocess resource [0x200000bd1:0xc23:0x0].0x0 (ffff8dfc4c55ae00) 00010000:00010000:3.0:1777502524.545656:0:205387:0:(ldlm_inodebits.c:76:ldlm_reprocess_inodebits_queue()) Hint 11 00010000:00010000:3.0:1777502524.545668:0:205387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: mdt-lustre-MDT0000_UUID lock: ffff8dfc77a17200/0x4ad31463d9c2df26 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0xc23:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x44a01000000000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf8453ac expref: 13 pid: 205369 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502524.545702:0:205387:0:(ldlm_inodebits.c:74:ldlm_reprocess_inodebits_queue()) --- Reprocess resource [0x200000bd2:0x227:0x0].0x0 (ffff8dfc59173000) 00010000:00010000:3.0:1777502524.545715:0:205387:0:(ldlm_inodebits.c:76:ldlm_reprocess_inodebits_queue()) Hint 8 00010000:00010000:3.0:1777502524.545719:0:205387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: mdt-lustre-MDT0000_UUID lock: ffff8dfd77e6d000/0x4ad31463d9c2e4f9 lrc: 0/0,0 mode: --/CR res: [0x200000bd2:0x227:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x44801000000000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf845614 expref: 12 pid: 187571 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502524.545745:0:205387:0:(ldlm_inodebits.c:74:ldlm_reprocess_inodebits_queue()) --- Reprocess resource [0x200000bd1:0xc22:0x0].0x0 (ffff8dfcbdf48100) 00010000:00010000:3.0:1777502524.545748:0:205387:0:(ldlm_inodebits.c:76:ldlm_reprocess_inodebits_queue()) Hint 1b 00010000:00010000:3.0:1777502524.545754:0:205387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: mdt-lustre-MDT0000_UUID lock: ffff8dfc77a17400/0x4ad31463d9c2de46 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0xc22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x44a01000000000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf845358 expref: 11 pid: 205372 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502524.545780:0:205387:0:(ldlm_inodebits.c:74:ldlm_reprocess_inodebits_queue()) --- Reprocess resource [0x200000bd1:0xc28:0x0].0x0 (ffff8dfc51877f00) 00010000:00010000:3.0:1777502524.545784:0:205387:0:(ldlm_inodebits.c:76:ldlm_reprocess_inodebits_queue()) Hint 1a 00010000:00010000:3.0:1777502524.545789:0:205387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: mdt-lustre-MDT0000_UUID lock: ffff8dfc77a16a00/0x4ad31463d9c2e1cd lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0xc28:0x0].0x0 bits 0x1a/0x0 rrc: 2 type: IBT gid 0 flags: 0x44a01000000000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf845493 expref: 10 pid: 187571 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502524.545815:0:205387:0:(ldlm_inodebits.c:74:ldlm_reprocess_inodebits_queue()) --- Reprocess resource [0x200000bd1:0xc24:0x0].0x0 (ffff8dfc4c55af00) 00010000:00010000:3.0:1777502524.545818:0:205387:0:(ldlm_inodebits.c:76:ldlm_reprocess_inodebits_queue()) Hint 1b 00010000:00010000:3.0:1777502524.545824:0:205387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: mdt-lustre-MDT0000_UUID lock: ffff8dfc77a17e00/0x4ad31463d9c2df42 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0xc24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x44a01000000000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf8453b3 expref: 10 pid: 188648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502524.545850:0:205387:0:(ldlm_inodebits.c:74:ldlm_reprocess_inodebits_queue()) --- Reprocess resource [0x200000bd1:0xc2a:0x0].0x0 (ffff8dfcbdf48200) 00010000:00010000:3.0:1777502524.545853:0:205387:0:(ldlm_inodebits.c:76:ldlm_reprocess_inodebits_queue()) Hint 1b 00010000:00010000:3.0:1777502524.545858:0:205387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: mdt-lustre-MDT0000_UUID lock: ffff8dfc85598400/0x4ad31463d9c2e458 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0xc2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x44a01000000000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf845581 expref: 9 pid: 188648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502524.545884:0:205387:0:(ldlm_inodebits.c:74:ldlm_reprocess_inodebits_queue()) --- Reprocess resource [0x200000bd1:0xc1f:0x0].0x0 (ffff8dfc4c55a600) 00010000:00010000:3.0:1777502524.545892:0:205387:0:(ldlm_inodebits.c:76:ldlm_reprocess_inodebits_queue()) Hint 11 00010000:00010000:3.0:1777502524.545914:0:205387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: mdt-lustre-MDT0000_UUID lock: ffff8dfc5765dc00/0x4ad31463d9c2dd97 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0xc1f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x44a01000000000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf845320 expref: 8 pid: 205369 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502524.545943:0:205387:0:(ldlm_inodebits.c:74:ldlm_reprocess_inodebits_queue()) --- Reprocess resource [0x200000007:0x1:0x0].0x0 (ffff8dfc51877100) 00010000:00010000:3.0:1777502524.545947:0:205387:0:(ldlm_inodebits.c:76:ldlm_reprocess_inodebits_queue()) Hint 13 00010000:00010000:3.0:1777502524.545953:0:205387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: mdt-lustre-MDT0000_UUID lock: ffff8dfc85599600/0x4ad31463d9c2e507 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x44a01000000000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf84561b expref: 8 pid: 186936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502524.545976:0:205387:0:(ldlm_lock.c:2680:ldlm_export_cancel_locks()) Export ffff8dfd66b1b800, canceled 9 locks, left on hash table 0. 00010000:00010000:3.0:1777502524.545998:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502524.546002:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502524.546007:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1777502524.546055:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d0000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0775d2ab-558e-41f1-82c0-14244d133390+4:2369:x1863837567212160:12345-192.168.202.27@tcp:39:umount.0 Request processed in 662us (771us total) trans 8589949329 rc 0/0 00000100:00100000:2.0:1777502524.546064:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 16658 00000100:00100000:2.0:1777502524.553761:0:3304:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.27@tcp (source: 12345-192.168.202.27@tcp) 00000100:00100000:3.0:1777502524.553818:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837567212416 00000100:00100000:3.0:1777502524.553826:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837567212416 00000100:00100000:3.0:1777502524.553871:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.27@tcp, seq: 393 00000100:00100000:3.0:1777502524.553879:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1f050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:c3865704-aa9b-444e-8420-b5af90b3c217+6:2369:x1863837567212416:12345-192.168.202.27@tcp:251:umount.0 00000020:00000080:3.0:1777502524.553917:0:186930:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x4ad31463d9b9e822 00000020:00010000:3.0:1777502524.553927:0:186930:0:(genops.c:1942:obd_stale_export_put()) Put export ffff8dfd66bff800: total 0 00000100:00080000:3.0:1777502524.553933:0:186930:0:(import.c:65:import_set_state_nolock()) ffff8dfd66bfd000 : changing import state from FULL to CLOSED 00000020:00010000:2.0:1777502524.553979:0:186925:0:(genops.c:1928:obd_stale_export_get()) Get export ffff8dfd66bff800: total 0 00000100:00100000:3.0:1777502524.553994:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1f050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:c3865704-aa9b-444e-8420-b5af90b3c217+3:2369:x1863837567212416:12345-192.168.202.27@tcp:251:umount.0 Request processed in 115us (240us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502524.554003:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.27@tcp, seq: 393 00010000:00010000:2.0:1777502524.554024:0:186925:0:(ldlm_lock.c:2652:ldlm_export_cancel_blocked_locks()) Export ffff8dfd66bff800, canceled 0 locks, left on hash table 1. 00010000:00010000:2.0:1777502524.554056:0:186925:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGS lock: ffff8dfc4e51f600/0x4ad31463d9b9e829 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 9 type: PLN flags: 0x44801000000000 nid: 192.168.202.27@tcp remote: 0xac3ccf43bf82a8c1 expref: 2 pid: 186930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502524.554074:0:186925:0:(ldlm_lock.c:2680:ldlm_export_cancel_locks()) Export ffff8dfd66bff800, canceled 1 locks, left on hash table 0. 00000020:00000080:2.0:1777502524.554091:0:186925:0:(genops.c:906:class_export_put()) final put ffff8dfd66bff800/c3865704-aa9b-444e-8420-b5af90b3c217 00010000:00010000:2.0:1777502524.554107:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502524.554110:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502524.554114:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000080:0.0:1777502524.554124:0:194113:0:(genops.c:865:class_export_destroy()) destroying export ffff8dfd66bff800/c3865704-aa9b-444e-8420-b5af90b3c217 for MGS 00000020:00000080:0.0:1777502524.554169:0:194113:0:(genops.c:1204:obd_zombie_import_free()) destroying import ffff8dfd66bfd000 for MGS 00010000:00010000:2.0:1777502524.881697:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1777502525.522103:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd509d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562080384:0@lo:400:kworker.0 00000100:00100000:3.0:1777502525.522171:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502525.522228:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a059180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3311:1863837562080640:0@lo:400:kworker.0 00000100:00100000:0.0:1777502525.522239:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562080768:0@lo:400:kworker.0 00000100:00100000:3.0:1777502525.522266:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502525.522317:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502525.522336:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3311:1863837562081152:0@lo:400:kworker.0 00000100:00100000:3.0:1777502525.522365:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502525.522372:0:205375:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562080640 00000100:00100000:0.0:1777502525.522380:0:205375:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562080640 00000100:00100000:3.0:1777502525.522410:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562080384 00000100:00100000:3.0:1777502525.522415:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562080384 00000100:00100000:3.0:1777502525.522462:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 394 00000100:00100000:0.0:1777502525.522464:0:205375:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562080768 00000100:00100000:2.0:1777502525.522469:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6fbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3310:1863837562080512:0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1777502525.522470:0:205375:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562080768 00000100:00100000:3.0:1777502525.522482:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1dc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3309:x1863837562080384:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502525.522526:0:205375:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 488 00000100:00100000:0.0:1777502525.522537:0:205375:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a05aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_007:lustre-MDT0000-mdtlov_UUID+5:3311:x1863837562080640:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502525.522552:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502525.522571:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1dc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3309:x1863837562080384:12345-0@lo:400:kworker.0 Request processed in 97us (399us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502525.522613:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 394 00000100:00100000:0.0:1777502525.522660:0:205375:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a05aa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_007:lustre-MDT0000-mdtlov_UUID+5:3311:x1863837562080640:12345-0@lo:400:kworker.0 Request processed in 116us (386us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502525.522674:0:205375:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 488 00000100:00100000:2.0:1777502525.522705:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-OST0000_UUID:3310:1863837562081024:0@lo:400:kworker.0 00000100:00100000:0.0:1777502525.522718:0:205375:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 489 00000100:00100000:0.0:1777502525.522729:0:205375:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f597b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_007:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562080768:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502525.522760:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502525.522810:0:205375:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f597b80 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_007:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562080768:12345-0@lo:400:kworker.0 Request processed in 79us (488us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502525.522818:0:205375:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 489 00000100:00080000:0.0:1777502525.522868:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000100:00080000:0.0:1777502525.522874:0:186932:0:(pinger.c:498:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:2.0:1777502525.522902:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562081152 00000100:00100000:2.0:1777502525.522909:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562081152 00000100:00100000:3.0:1777502525.522912:0:189738:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562080512 00000100:00100000:3.0:1777502525.522918:0:189738:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562080512 00000100:00100000:0.0:1777502525.522965:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f597480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562081280:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1777502525.522993:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562081024 00000100:00100000:2.0:1777502525.522995:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562081024 00000100:00100000:3.0:1777502525.523003:0:189738:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 274 00000100:00100000:3.0:1777502525.523023:0:189738:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6faa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3310:x1863837562080512:12345-0@lo:13:osp-pre-0-0.0 00000100:00100000:0.0:1777502525.523026:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502525.523043:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 470 00000100:00100000:2.0:1777502525.523053:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a059500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0001_UUID+5:3311:x1863837562081152:12345-0@lo:400:kworker.0 00002000:00000024:3.0:1777502525.523057:0:189738:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 33488896 pending 0 free 3831496704 avail 3829399552 00000100:00100000:0.0:1777502525.523094:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562080768:0@lo:400:kworker.0 00000100:00100000:2.0:1777502525.523129:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a059500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-OST0001_UUID+5:3311:x1863837562081152:12345-0@lo:400:kworker.0 Request processed in 76us (760us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502525.523143:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 470 00000100:00100000:3.0:1777502525.523145:0:189738:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6faa00 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3310:x1863837562080512:12345-0@lo:13:osp-pre-0-0.0 Request processed in 129us (594us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502525.523158:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 471 00000100:00100000:3.0:1777502525.523158:0:189738:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 274 00000100:00100000:0.0:1777502525.523182:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6fb480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0000_UUID+5:3310:x1863837562081024:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502525.523191:0:189738:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562081280 00000100:00100000:3.0:1777502525.523195:0:189738:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562081280 00000100:00100000:3.0:1777502525.523250:0:189738:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 275 00000100:00100000:0.0:1777502525.523257:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6fb480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:lustre-MDT0000-lwp-OST0000_UUID+5:3310:x1863837562081024:12345-0@lo:400:kworker.0 Request processed in 81us (497us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502525.523264:0:189738:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4abbe050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562081280:12345-0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1777502525.523270:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 471 00000100:00100000:2.0:1777502525.523281:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd509d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562080384:0@lo:400:kworker.0 00002000:00000024:3.0:1777502525.523287:0:189738:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 33488896 pending 0 free 3801088000 avail 3798990848 00000100:00100000:2.0:1777502525.523297:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3309:1863837562080896:0@lo:400:kworker.0 00000100:00100000:0.0:1777502525.523343:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a059180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3311:1863837562080640:0@lo:400:kworker.0 00000100:00100000:3.0:1777502525.523355:0:189738:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4abbe050 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3312:x1863837562081280:12345-0@lo:13:osp-pre-1-0.0 Request processed in 88us (328us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502525.523361:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502525.523363:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:3311:1863837562081152:0@lo:400:kworker.0 00000100:00100000:3.0:1777502525.523380:0:189738:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 275 00000100:00100000:0.0:1777502525.523437:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f597480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-mdtlov_UUID:3312:1863837562081280:0@lo:13:osp-pre-1-0.0 00000100:00100000:0.0:1777502525.523639:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562080896 00000100:00100000:0.0:1777502525.523643:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562080896 00000100:00100000:0.0:1777502525.523679:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 472 00000100:00100000:0.0:1777502525.523690:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6f9500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:lustre-MDT0000-lwp-MDT0000_UUID+5:3309:x1863837562080896:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502525.523749:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6f9500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:lustre-MDT0000-lwp-MDT0000_UUID+5:3309:x1863837562080896:12345-0@lo:400:kworker.0 Request processed in 61us (403us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502525.523756:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 472 00000100:00100000:0.0:1777502525.523805:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3309:1863837562080896:0@lo:400:kworker.0 00000100:00100000:3.0:1777502525.523858:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6fbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-mdtlov_UUID:3310:1863837562080512:0@lo:13:osp-pre-0-0.0 00000100:00100000:3.0:1777502525.523888:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:lustre-MDT0000-lwp-OST0000_UUID:3310:1863837562081024:0@lo:400:kworker.0 00000001:00100000:3.0:1777502525.544570:0:7467:0:(tgt_lastrcvd.c:568:tgt_cb_new_client()) lustre-OST0001: committing for initial connect of 0775d2ab-558e-41f1-82c0-14244d133390 00000020:00000080:3.0:1777502525.544592:0:7467:0:(genops.c:906:class_export_put()) final put ffff8dfd4fd53000/0775d2ab-558e-41f1-82c0-14244d133390 00000020:00000080:2.0:1777502525.544615:0:194113:0:(genops.c:865:class_export_destroy()) destroying export ffff8dfd4fd53000/0775d2ab-558e-41f1-82c0-14244d133390 for lustre-OST0001 00000020:00000080:2.0:1777502525.544654:0:194113:0:(genops.c:1204:obd_zombie_import_free()) destroying import ffff8dfd4fd57000 for lustre-OST0001 00000001:00100000:3.0:1777502525.546204:0:6599:0:(tgt_lastrcvd.c:568:tgt_cb_new_client()) lustre-OST0000: committing for initial connect of 0775d2ab-558e-41f1-82c0-14244d133390 00000020:00000080:3.0:1777502525.546209:0:6599:0:(genops.c:906:class_export_put()) final put ffff8dfd4fd51800/0775d2ab-558e-41f1-82c0-14244d133390 00000020:00000080:3.0:1777502525.546248:0:173604:0:(genops.c:865:class_export_destroy()) destroying export ffff8dfd4fd51800/0775d2ab-558e-41f1-82c0-14244d133390 for lustre-OST0000 00000020:00000080:3.0:1777502525.546285:0:173604:0:(genops.c:1204:obd_zombie_import_free()) destroying import ffff8dfd4fd50000 for lustre-OST0000 00000001:02000400:2.0:1777502525.653011:0:221515:0:(debug.c:645:libcfs_debug_mark_buffer()) DEBUG MARKER: === sanityn: finish cleanup 18:42:04 (1777502524) === 00010000:00010000:0.0:1777502525.906672:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:01000004:2.0:1777502526.404077:0:221612:0:(tgt_mount.c:2147:server_umount_begin()) umount -f 00000020:01000004:3.0:1777502526.409775:0:221612:0:(tgt_mount.c:2017:server_put_super()) server put_super lustre-MDT0000 00010000:00010000:3.0:1777502526.409825:0:221612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6e00/0x4ad31463d9b9e55f lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 8 type: PLN flags: 0x11000000000000 nid: local remote: 0x4ad31463d9b9e566 expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.409839:0:221612:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6e00/0x4ad31463d9b9e55f lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 8 type: PLN flags: 0x1000000000000 nid: local remote: 0x4ad31463d9b9e566 expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.409847:0:221612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6e00/0x4ad31463d9b9e55f lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 8 type: PLN flags: 0x11000400000000 nid: local remote: 0x4ad31463d9b9e566 expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.409855:0:221612:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6e00/0x4ad31463d9b9e55f lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 8 type: PLN flags: 0x11000400000000 nid: local remote: 0x4ad31463d9b9e566 expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.409877:0:221612:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 10000000:01000000:3.0:1777502526.409882:0:221612:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00010000:00010000:3.0:1777502526.409915:0:205387:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1777502526.409922:0:205387:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6e00/0x4ad31463d9b9e55f lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 8 type: PLN flags: 0x1000400000000 nid: local remote: 0x4ad31463d9b9e566 expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.409930:0:205387:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8dfd80fd6e00 already unused, calling callback (ffffffffc19fbc20) 10000000:00010000:3.0:1777502526.409934:0:205387:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6e00/0x4ad31463d9b9e55f lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 8 type: PLN flags: 0x1000400000000 nid: local remote: 0x4ad31463d9b9e566 expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.409952:0:205387:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6e00/0x4ad31463d9b9e55f lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 8 type: PLN flags: 0x1008400000000 nid: local remote: 0x4ad31463d9b9e566 expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1777502526.409960:0:205387:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6e00/0x4ad31463d9b9e55f lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 8 type: PLN flags: 0x1009400000000 nid: local remote: 0x4ad31463d9b9e566 expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1777502526.409965:0:205387:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:3.0:1777502526.409968:0:205387:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 10000000:01000000:3.0:1777502526.409972:0:205387:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 00010000:00010000:3.0:1777502526.410007:0:205387:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6e00/0x4ad31463d9b9e55f lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 8 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9e566 expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.410014:0:205387:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1777502526.410042:0:205387:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6e00/0x4ad31463d9b9e55f lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 8 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9e566 expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.410050:0:205387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6e00/0x4ad31463d9b9e55f lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 8 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9e566 expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1777502526.410060:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562081408:0@lo:103:ldlm_bl.0 00010000:00010000:3.0:1777502526.410066:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502526.410070:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502526.410074:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502526.410097:0:221612:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:3.0:1777502526.410102:0:221612:0:(client.c:3101:ptlrpc_free_committed()) lustre-MDT0000-lwp-MDT0000: committing for last_committed 0 gen 3 00000100:00080000:3.0:1777502526.410109:0:221612:0:(pinger.c:396:ptlrpc_pinger_del_import()) removing pingable import lustre-MDT0000-lwp-MDT0000_UUID->lustre-MDT0000_UUID 00000100:00100000:0.0:1777502526.410120:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:3.0:1777502526.410125:0:221612:0:(import.c:65:import_set_state_nolock()) ffff8dfd4acec800 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1777502526.410164:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a059880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-MDT0000_UUID:3308:1863837562081536:0@lo:39:umount.0 00000100:00100000:0.0:1777502526.410196:0:205389:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562081408 00000100:00100000:0.0:1777502526.410202:0:205389:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562081408 00000100:00100000:2.0:1777502526.410209:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502526.410255:0:205389:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 300 00000100:00100000:0.0:1777502526.410265:0:205389:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f597480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_006:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3309:x1863837562081408:12345-0@lo:103:ldlm_bl.0 00000100:00100000:2.0:1777502526.410265:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562081536 00000100:00100000:2.0:1777502526.410270:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562081536 00010000:00010000:0.0:1777502526.410280:0:205389:0:(ldlm_lockd.c:1747:ldlm_request_cancel()) ### server-side cancel handler START: 1 locks, starting at 0 00010000:00010000:0.0:1777502526.410298:0:205389:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGS lock: ffff8dfd58a72c00/0x4ad31463d9b9e566 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 9 type: PLN flags: 0x44801000000000 nid: 0@lo remote: 0x4ad31463d9b9e55f expref: 18 pid: 186931 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1777502526.410312:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16659 00010000:00010000:0.0:1777502526.410313:0:205389:0:(ldlm_lockd.c:1808:ldlm_request_cancel()) ### server-side cancel handler END 00000100:00100000:2.0:1777502526.410321:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6fbb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-MDT0000_UUID+5:3308:x1863837562081536:12345-0@lo:39:umount.0 00000020:00000080:2.0:1777502526.410348:0:186937:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x4ad31463d9b9e615 00000100:00100000:0.0:1777502526.410349:0:205389:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f597480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_006:d37a34a7-24bb-490e-b7a8-76d369c5604f+17:3309:x1863837562081408:12345-0@lo:103:ldlm_bl.0 Request processed in 85us (228us total) trans 0 rc 0/0 00000100:00080000:2.0:1777502526.410356:0:186937:0:(import.c:65:import_set_state_nolock()) ffff8dfd4ed72800 : changing import state from FULL to CLOSED 00000100:00100000:0.0:1777502526.410359:0:205389:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 300 00000100:00100000:0.0:1777502526.410405:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562081408:0@lo:103:ldlm_bl.0 00000100:00100000:2.0:1777502526.410416:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6fbb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:lustre-MDT0000-lwp-MDT0000_UUID+1:3308:x1863837562081536:12345-0@lo:39:umount.0 Request processed in 97us (210us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502526.410423:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16659 00000020:00000080:2.0:1777502526.410433:0:186937:0:(genops.c:906:class_export_put()) final put ffff8dfd4ed77800/lustre-MDT0000-lwp-MDT0000_UUID 00010000:00010000:0.0:1777502526.410441:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000080:0.0:1777502526.410455:0:194113:0:(genops.c:865:class_export_destroy()) destroying export ffff8dfd4ed77800/lustre-MDT0000-lwp-MDT0000_UUID for lustre-MDT0000 00000100:00080000:2.0:1777502526.410475:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd4acec800 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1777502526.410486:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a059880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-MDT0000_UUID:3308:1863837562081536:0@lo:39:umount.0 00000100:00080000:2.0:1777502526.410519:0:221612:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000020:00000080:0.0:1777502526.410521:0:194113:0:(genops.c:1204:obd_zombie_import_free()) destroying import ffff8dfd4ed72800 for lustre-MDT0000 00000100:00100000:2.0:1777502526.410522:0:221612:0:(client.c:3101:ptlrpc_free_committed()) lustre-MDT0000-lwp-MDT0000: committing for last_committed 0 gen 4 00000100:00100000:2.0:1777502526.410528:0:221612:0:(import.c:321:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00010000:00010000:2.0:1777502526.411300:0:221612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7200/0x4ad31463d9b9e4c5 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 7 type: PLN flags: 0x11000000000000 nid: local remote: 0x4ad31463d9b9e4cc expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502526.411309:0:221612:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7200/0x4ad31463d9b9e4c5 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 7 type: PLN flags: 0x1000000000000 nid: local remote: 0x4ad31463d9b9e4cc expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502526.411314:0:221612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7200/0x4ad31463d9b9e4c5 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 7 type: PLN flags: 0x11000400000000 nid: local remote: 0x4ad31463d9b9e4cc expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502526.411318:0:221612:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7200/0x4ad31463d9b9e4c5 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 7 type: PLN flags: 0x11000400000000 nid: local remote: 0x4ad31463d9b9e4cc expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502526.411337:0:221612:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1777502526.411344:0:221612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6c00/0x4ad31463d9b9e814 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x4ad31463d9b9e81b expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502526.411350:0:221612:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6c00/0x4ad31463d9b9e814 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x4ad31463d9b9e81b expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502526.411352:0:188362:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1777502526.411358:0:188362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7200/0x4ad31463d9b9e4c5 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 7 type: PLN flags: 0x1000400000000 nid: local remote: 0x4ad31463d9b9e4cc expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502526.411376:0:188362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8dfd80fd7200 already unused, calling callback (ffffffffc19fbc20) 00010000:00010000:2.0:1777502526.411378:0:221612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6c00/0x4ad31463d9b9e814 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x4ad31463d9b9e81b expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1777502526.411380:0:188362:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7200/0x4ad31463d9b9e4c5 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 7 type: PLN flags: 0x1000400000000 nid: local remote: 0x4ad31463d9b9e4cc expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502526.411396:0:221612:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6c00/0x4ad31463d9b9e814 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x4ad31463d9b9e81b expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502526.411397:0:188362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7200/0x4ad31463d9b9e4c5 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 7 type: PLN flags: 0x1008400000000 nid: local remote: 0x4ad31463d9b9e4cc expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1777502526.411414:0:188362:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7200/0x4ad31463d9b9e4c5 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 7 type: PLN flags: 0x1009400000000 nid: local remote: 0x4ad31463d9b9e4cc expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502526.411422:0:221612:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1777502526.411428:0:221612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6800/0x4ad31463d9b9ea8a lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 4 type: PLN flags: 0x11000000000000 nid: local remote: 0x4ad31463d9b9ea91 expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1777502526.411432:0:188362:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:0.0:1777502526.411439:0:188362:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-MDT0000: stopping, won't requeue 00010000:00010000:2.0:1777502526.411440:0:221612:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6800/0x4ad31463d9b9ea8a lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 4 type: PLN flags: 0x1000000000000 nid: local remote: 0x4ad31463d9b9ea91 expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502526.411446:0:221612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6800/0x4ad31463d9b9ea8a lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 4 type: PLN flags: 0x11000400000000 nid: local remote: 0x4ad31463d9b9ea91 expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502526.411450:0:221612:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6800/0x4ad31463d9b9ea8a lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 4 type: PLN flags: 0x11000400000000 nid: local remote: 0x4ad31463d9b9ea91 expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502526.411457:0:188362:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7200/0x4ad31463d9b9e4c5 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 7 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9e4cc expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502526.411463:0:188362:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1777502526.411465:0:221612:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:2.0:1777502526.411471:0:221612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7e00/0x4ad31463d9b9e4b7 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x5:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x4ad31463d9b9e4be expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502526.411476:0:221612:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7e00/0x4ad31463d9b9e4b7 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x5:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x4ad31463d9b9e4be expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.411476:0:205387:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1777502526.411480:0:188362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7200/0x4ad31463d9b9e4c5 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 7 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9e4cc expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.411485:0:205387:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6c00/0x4ad31463d9b9e814 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x4ad31463d9b9e81b expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502526.411486:0:221612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7e00/0x4ad31463d9b9e4b7 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x5:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x4ad31463d9b9e4be expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502526.411499:0:188362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7200/0x4ad31463d9b9e4c5 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 7 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9e4cc expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502526.411513:0:188362:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1777502526.411519:0:188362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6800/0x4ad31463d9b9ea8a lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 4 type: PLN flags: 0x1000400000000 nid: local remote: 0x4ad31463d9b9ea91 expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502526.411530:0:221612:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7e00/0x4ad31463d9b9e4b7 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x5:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x4ad31463d9b9e4be expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.411531:0:205387:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8dfd80fd6c00 already unused, calling callback (ffffffffc19fbc20) 00010000:00010000:0.0:1777502526.411532:0:188362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8dfd80fd6800 already unused, calling callback (ffffffffc19fbc20) 10000000:00010000:3.0:1777502526.411537:0:205387:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6c00/0x4ad31463d9b9e814 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x4ad31463d9b9e81b expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1777502526.411538:0:188362:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6800/0x4ad31463d9b9ea8a lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 4 type: PLN flags: 0x1000400000000 nid: local remote: 0x4ad31463d9b9ea91 expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1777502526.411558:0:221612:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1777502526.411561:0:188362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6800/0x4ad31463d9b9ea8a lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0x4ad31463d9b9ea91 expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.411561:0:205387:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6c00/0x4ad31463d9b9e814 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0x4ad31463d9b9e81b expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1777502526.411562:0:221612:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-MDT0000 10000000:01000000:2.0:1777502526.411568:0:221612:0:(mgc_request.c:547:config_log_end()) end config log lustre-MDT0000 (0) 10000000:00010000:3.0:1777502526.411604:0:205387:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6c00/0x4ad31463d9b9e814 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0x4ad31463d9b9e81b expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:01000004:2.0:1777502526.411605:0:221612:0:(tgt_mount.c:2067:server_put_super()) stopping lustre-MDT0000 00000020:01000000:2.0:1777502526.411610:0:221612:0:(obd_config.c:2475:class_manual_cleanup()) Manual cleanup of lustre-MDT0000 (flags='F') 10000000:01000000:3.0:1777502526.411612:0:205387:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00000020:00000080:2.0:1777502526.411613:0:221612:0:(obd_config.c:1470:class_process_config()) processing cmd: cf004 10000000:01000000:3.0:1777502526.411616:0:205387:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-mdtir: stopping, won't requeue 00000020:00000080:2.0:1777502526.411620:0:221612:0:(obd_config.c:955:class_cleanup()) lustre-MDT0000: forcing exports to disconnect: 2/7 10000000:01000000:3.0:1777502526.411620:0:205387:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-mdtir 00000020:00080000:2.0:1777502526.411627:0:221612:0:(genops.c:1840:print_export_data()) lustre-MDT0000: ACTIVE ffff8dfd6e957800 lustre-MDT0000-lwp-OST0001_UUID 0@lo 4 (0 0 0) 0 0 0 0: 0000000000000000 0 stale:0 00000020:00080000:2.0:1777502526.411632:0:221612:0:(genops.c:1840:print_export_data()) lustre-MDT0000: ACTIVE ffff8dfd42ae2800 lustre-MDT0000-lwp-OST0000_UUID 0@lo 4 (0 0 0) 0 0 0 0: 0000000000000000 0 stale:0 00000020:00080000:2.0:1777502526.411638:0:221612:0:(genops.c:1840:print_export_data()) lustre-MDT0000: UNLINKED ffff8dfd66b1b800 0775d2ab-558e-41f1-82c0-14244d133390 192.168.202.27@tcp 3 (0 2 0) 1 0 0 0: 0000000000000000 8589949328 stale:0 00000020:00080000:2.0:1777502526.411642:0:221612:0:(genops.c:1574:class_disconnect_exports()) OBD device 5 (ffff8dfd4f2e19c0) has exports, disconnecting them 00010000:00010000:3.0:1777502526.411642:0:205387:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6c00/0x4ad31463d9b9e814 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9e81b expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00080000:2.0:1777502526.411647:0:221612:0:(genops.c:1549:class_disconnect_export_list()) lustre-MDT0000: disconnecting export at 0@lo (ffff8dfd6e957800), last request at 1777502525 00000020:00000080:2.0:1777502526.411654:0:221612:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x4ad31463d9b9e685 00010000:00010000:3.0:1777502526.411654:0:205387:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00000100:00080000:2.0:1777502526.411668:0:221612:0:(import.c:65:import_set_state_nolock()) ffff8dfd6e953000 : changing import state from FULL to CLOSED 00010000:00010000:3.0:1777502526.411678:0:205387:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6c00/0x4ad31463d9b9e814 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9e81b expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.411697:0:205387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6c00/0x4ad31463d9b9e814 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9e81b expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1777502526.411718:0:188362:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6800/0x4ad31463d9b9ea8a lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0x4ad31463d9b9ea91 expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.411718:0:205387:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00000020:00080000:2.0:1777502526.411724:0:221612:0:(genops.c:1554:class_disconnect_export_list()) disconnected export at 0@lo (ffff8dfd6e957800): rc 0 00010000:00010000:3.0:1777502526.411725:0:205387:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7e00/0x4ad31463d9b9e4b7 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x5:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x4ad31463d9b9e4be expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00000080:2.0:1777502526.411730:0:221612:0:(genops.c:906:class_export_put()) final put ffff8dfd6e957800/lustre-MDT0000-lwp-OST0001_UUID 10000000:01000000:0.0:1777502526.411731:0:188362:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:0.0:1777502526.411737:0:188362:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:0.0:1777502526.411741:0:188362:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:3.0:1777502526.411742:0:205387:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8dfd80fd7e00 already unused, calling callback (ffffffffc19fbc20) 10000000:00010000:3.0:1777502526.411746:0:205387:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7e00/0x4ad31463d9b9e4b7 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x5:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x4ad31463d9b9e4be expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00080000:2.0:1777502526.411749:0:221612:0:(genops.c:1549:class_disconnect_export_list()) lustre-MDT0000: disconnecting export at 0@lo (ffff8dfd42ae2800), last request at 1777502525 00000020:00000080:2.0:1777502526.411755:0:221612:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x4ad31463d9b9e60e 00010000:00010000:0.0:1777502526.411762:0:188362:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6800/0x4ad31463d9b9ea8a lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9ea91 expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.411764:0:205387:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7e00/0x4ad31463d9b9e4b7 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x5:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0x4ad31463d9b9e4be expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1777502526.411765:0:221612:0:(import.c:65:import_set_state_nolock()) ffff8dfd4f2f4000 : changing import state from FULL to CLOSED 00000020:00080000:2.0:1777502526.411779:0:221612:0:(genops.c:1554:class_disconnect_export_list()) disconnected export at 0@lo (ffff8dfd42ae2800): rc 0 00010000:00010000:0.0:1777502526.411781:0:188362:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00000020:00000080:2.0:1777502526.411785:0:221612:0:(genops.c:906:class_export_put()) final put ffff8dfd42ae2800/lustre-MDT0000-lwp-OST0000_UUID 10000000:00010000:3.0:1777502526.411786:0:205387:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7e00/0x4ad31463d9b9e4b7 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x5:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0x4ad31463d9b9e4be expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502526.411792:0:188362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6800/0x4ad31463d9b9ea8a lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9ea91 expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1777502526.411793:0:205387:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x5:0x0].0x0 (lustre) 00000004:00000080:2.0:1777502526.411795:0:221612:0:(mdd_device.c:2302:mdd_iocontrol()) lustre-MDD0000: cmd=400866e7 len=0 karg=ffffb31802a67ae0 10000000:01000000:3.0:1777502526.411800:0:205387:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-barrier: stopping, won't requeue 10000000:01000000:3.0:1777502526.411804:0:205387:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-barrier 00010000:00010000:0.0:1777502526.411807:0:188362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd6800/0x4ad31463d9b9ea8a lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9ea91 expref: -99 pid: 186933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1777502526.411813:0:221612:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00010000:00010000:0.0:1777502526.411818:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502526.411820:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1777502526.411822:0:221612:0:(client.c:3101:ptlrpc_free_committed()) lustre-OST0000-osc-MDT0000: committing for last_committed 8589937466 gen 3 00010000:00010000:0.0:1777502526.411823:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502526.411826:0:205387:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7e00/0x4ad31463d9b9e4b7 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x5:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9e4be expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502526.411831:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000004:00080000:2.0:1777502526.411833:0:221612:0:(osp_dev.c:1629:osp_import_event()) got inactive 00010000:00010000:3.0:1777502526.411834:0:205387:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1777502526.411843:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f595c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562081664:0@lo:103:ldlm_bl.0 00010000:00010000:3.0:1777502526.411845:0:205387:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7e00/0x4ad31463d9b9e4b7 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x5:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9e4be expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.411853:0:205387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.127@tcp lock: ffff8dfd80fd7e00/0x4ad31463d9b9e4b7 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x5:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x4ad31463d9b9e4be expref: -99 pid: 186918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1777502526.411871:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1777502526.411875:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00010000:00010000:3.0:1777502526.411876:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502526.411882:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502526.411903:0:221612:0:(pinger.c:396:ptlrpc_pinger_del_import()) removing pingable import lustre-MDT0000-mdtlov_UUID->lustre-OST0000_UUID 00000100:00100000:0.0:1777502526.411908:0:3310:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1777502526.411915:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562082048:0@lo:103:ldlm_bl.0 00000100:00080000:2.0:1777502526.411915:0:221612:0:(import.c:65:import_set_state_nolock()) ffff8dfd7ec9c000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:2.0:1777502526.411931:0:221612:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1777502526.411933:0:221612:0:(client.c:3101:ptlrpc_free_committed()) lustre-OST0001-osc-MDT0000: committing for last_committed 8589937045 gen 3 00000004:00080000:2.0:1777502526.411941:0:221612:0:(osp_dev.c:1629:osp_import_event()) got inactive 00000100:00100000:0.0:1777502526.411946:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502526.411976:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a058380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562081792:0@lo:103:ldlm_bl.0 00000100:00100000:0.0:1777502526.412002:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502526.412008:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6f9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:3308:1863837562082176:0@lo:9:umount.0 00000100:00100000:0.0:1777502526.412022:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:0.0:1777502526.412030:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f594e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562081920:0@lo:103:ldlm_bl.0 00000100:00100000:2.0:1777502526.412045:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502526.412062:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502526.412077:0:205376:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562082176 00000100:00100000:3.0:1777502526.412083:0:205376:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562082176 00000100:00080000:2.0:1777502526.412086:0:221612:0:(pinger.c:396:ptlrpc_pinger_del_import()) removing pingable import lustre-MDT0000-mdtlov_UUID->lustre-OST0001_UUID 00000100:00080000:2.0:1777502526.412099:0:221612:0:(import.c:65:import_set_state_nolock()) ffff8dfd430e6800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:0.0:1777502526.412101:0:186922:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562081664 00000100:00100000:0.0:1777502526.412105:0:186922:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562081664 00000100:00100000:2.0:1777502526.412130:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6f9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:3308:1863837562082304:0@lo:9:umount.0 00000100:00100000:3.0:1777502526.412131:0:205376:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 6850 00000100:00100000:0.0:1777502526.412138:0:186922:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 301 00000100:00100000:3.0:1777502526.412144:0:205376:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6f9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_008:lustre-MDT0000-mdtlov_UUID+5:3308:x1863837562082176:12345-0@lo:9:umount.0 00000100:00100000:0.0:1777502526.412150:0:186922:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f594000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_001:d37a34a7-24bb-490e-b7a8-76d369c5604f+17:3310:x1863837562081664:12345-0@lo:103:ldlm_bl.0 00010000:00010000:0.0:1777502526.412159:0:186922:0:(ldlm_lockd.c:1747:ldlm_request_cancel()) ### server-side cancel handler START: 1 locks, starting at 0 00000020:00000080:3.0:1777502526.412164:0:205376:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x4ad31463d9b9e693 00000100:00100000:2.0:1777502526.412166:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:3.0:1777502526.412176:0:205376:0:(import.c:65:import_set_state_nolock()) ffff8dfd66bfe000 : changing import state from FULL to CLOSED 00010000:00010000:0.0:1777502526.412180:0:186922:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGS lock: ffff8dfd58a72400/0x4ad31463d9b9e4cc lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 8 type: PLN flags: 0x44801000000000 nid: 0@lo remote: 0x4ad31463d9b9e4c5 expref: 17 pid: 186931 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502526.412191:0:186922:0:(ldlm_lockd.c:1808:ldlm_request_cancel()) ### server-side cancel handler END 00000100:00100000:2.0:1777502526.412204:0:205368:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562082304 00000001:00000004:3.0:1777502526.412205:0:205376:0:(tgt_lastrcvd.c:732:tgt_server_data_update()) lustre-OST0000_UUID: mount_count is 2, last_transno is 8589937482 00000100:00100000:2.0:1777502526.412208:0:205368:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562082304 00000100:00100000:0.0:1777502526.412226:0:186922:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f594000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_001:d37a34a7-24bb-490e-b7a8-76d369c5604f+16:3310:x1863837562081664:12345-0@lo:103:ldlm_bl.0 Request processed in 81us (352us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502526.412234:0:186922:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 301 00000100:00100000:2.0:1777502526.412237:0:205368:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 6851 00000100:00100000:2.0:1777502526.412245:0:205368:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6fa300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_006:lustre-MDT0000-mdtlov_UUID+5:3308:x1863837562082304:12345-0@lo:9:umount.0 00000100:00100000:0.0:1777502526.412252:0:186922:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562082048 00000100:00100000:0.0:1777502526.412256:0:186922:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562082048 00000020:00000080:2.0:1777502526.412259:0:205368:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x4ad31463d9b9e69a 00000100:00080000:2.0:1777502526.412266:0:205368:0:(import.c:65:import_set_state_nolock()) ffff8dfd46532800 : changing import state from FULL to CLOSED 00000100:00100000:0.0:1777502526.412281:0:186922:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562081792 00000100:00100000:0.0:1777502526.412284:0:186922:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562081792 00000001:00000004:2.0:1777502526.412285:0:205368:0:(tgt_lastrcvd.c:732:tgt_server_data_update()) lustre-OST0001_UUID: mount_count is 2, last_transno is 8589937082 00000100:00100000:0.0:1777502526.412306:0:186922:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562081920 00000100:00100000:0.0:1777502526.412309:0:186922:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562081920 00000100:00100000:0.0:1777502526.412334:0:186922:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 302 00000100:00100000:0.0:1777502526.412340:0:186922:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f596a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_001:d37a34a7-24bb-490e-b7a8-76d369c5604f+18:3310:x1863837562082048:12345-0@lo:103:ldlm_bl.0 00010000:00010000:0.0:1777502526.412348:0:186922:0:(ldlm_lockd.c:1747:ldlm_request_cancel()) ### server-side cancel handler START: 1 locks, starting at 0 00010000:00010000:0.0:1777502526.412357:0:186922:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGS lock: ffff8dfd58a73800/0x4ad31463d9b9e4be lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x5:0x0].0x0 rrc: 3 type: PLN flags: 0x44801000000000 nid: 0@lo remote: 0x4ad31463d9b9e4b7 expref: 18 pid: 186929 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502526.412371:0:186922:0:(ldlm_lockd.c:1808:ldlm_request_cancel()) ### server-side cancel handler END 00000100:00100000:0.0:1777502526.412396:0:186922:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f596a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_001:d37a34a7-24bb-490e-b7a8-76d369c5604f+17:3310:x1863837562082048:12345-0@lo:103:ldlm_bl.0 Request processed in 56us (451us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502526.412402:0:186922:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 302 00000100:00100000:0.0:1777502526.412418:0:186922:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 303 00000100:00100000:0.0:1777502526.412426:0:186922:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f595180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_001:d37a34a7-24bb-490e-b7a8-76d369c5604f+16:3311:x1863837562081792:12345-0@lo:103:ldlm_bl.0 00010000:00010000:0.0:1777502526.412433:0:186922:0:(ldlm_lockd.c:1747:ldlm_request_cancel()) ### server-side cancel handler START: 1 locks, starting at 0 00010000:00010000:0.0:1777502526.412444:0:186922:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGS lock: ffff8dfc4e51ee00/0x4ad31463d9b9e81b lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x44801000000000 nid: 0@lo remote: 0x4ad31463d9b9e814 expref: 16 pid: 186929 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502526.412465:0:186922:0:(ldlm_lockd.c:1808:ldlm_request_cancel()) ### server-side cancel handler END 00000100:00100000:0.0:1777502526.412493:0:186922:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f595180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_001:d37a34a7-24bb-490e-b7a8-76d369c5604f+15:3311:x1863837562081792:12345-0@lo:103:ldlm_bl.0 Request processed in 70us (490us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502526.412499:0:186922:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 303 00000100:00100000:0.0:1777502526.412512:0:186922:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 304 00000100:00100000:0.0:1777502526.412518:0:186922:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd80180c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_001:d37a34a7-24bb-490e-b7a8-76d369c5604f+14:3311:x1863837562081920:12345-0@lo:103:ldlm_bl.0 00010000:00010000:0.0:1777502526.412525:0:186922:0:(ldlm_lockd.c:1747:ldlm_request_cancel()) ### server-side cancel handler START: 1 locks, starting at 0 00010000:00010000:0.0:1777502526.412532:0:186922:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGS lock: ffff8dfd67664400/0x4ad31463d9b9ea91 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 5 type: PLN flags: 0x44801000000000 nid: 0@lo remote: 0x4ad31463d9b9ea8a expref: 14 pid: 186930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1777502526.412542:0:186922:0:(ldlm_lockd.c:1808:ldlm_request_cancel()) ### server-side cancel handler END 00000100:00100000:0.0:1777502526.412565:0:186922:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd80180c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cn00_001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562081920:12345-0@lo:103:ldlm_bl.0 Request processed in 47us (504us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502526.412571:0:186922:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 304 00000020:00000080:0.0:1777502526.412668:0:173604:0:(genops.c:865:class_export_destroy()) destroying export ffff8dfd6e957800/lustre-MDT0000-lwp-OST0001_UUID for lustre-MDT0000 00000020:00000080:0.0:1777502526.412727:0:173604:0:(genops.c:865:class_export_destroy()) destroying export ffff8dfd42ae2800/lustre-MDT0000-lwp-OST0000_UUID for lustre-MDT0000 00000020:00000080:0.0:1777502526.412763:0:173604:0:(genops.c:1204:obd_zombie_import_free()) destroying import ffff8dfd6e953000 for lustre-MDT0000 00000100:00100000:2.0:1777502526.412768:0:205368:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6fa300 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_006:lustre-MDT0000-mdtlov_UUID+2:3308:x1863837562082304:12345-0@lo:9:umount.0 Request processed in 523us (602us total) trans 0 rc 0/0 00000020:00000080:0.0:1777502526.412769:0:173604:0:(genops.c:1204:obd_zombie_import_free()) destroying import ffff8dfd4f2f4000 for lustre-MDT0000 00000100:00100000:2.0:1777502526.412780:0:205368:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 6851 00010000:00010000:0.0:1777502526.412788:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502526.412793:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1777502526.412797:0:205376:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6f9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost00_008:lustre-MDT0000-mdtlov_UUID+2:3308:x1863837562082176:12345-0@lo:9:umount.0 Request processed in 654us (749us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502526.412807:0:205376:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 6850 00000100:00080000:2.0:1777502526.412822:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd430e6800 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:0.0:1777502526.412828:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f594e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562081920:0@lo:103:ldlm_bl.0 00000004:00080000:2.0:1777502526.412832:0:3308:0:(osp_dev.c:1613:osp_import_event()) got disconnected 00000004:00080000:2.0:1777502526.412835:0:3308:0:(osp_dev.c:1629:osp_import_event()) got inactive 00000100:00100000:2.0:1777502526.412839:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6f9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:3308:1863837562082304:0@lo:9:umount.0 00000100:00100000:0.0:1777502526.412841:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a058380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562081792:0@lo:103:ldlm_bl.0 00000100:00080000:2.0:1777502526.412855:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd7ec9c000 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000004:00080000:2.0:1777502526.412868:0:3308:0:(osp_dev.c:1613:osp_import_event()) got disconnected 00000004:00080000:2.0:1777502526.412871:0:3308:0:(osp_dev.c:1629:osp_import_event()) got inactive 00000100:00100000:2.0:1777502526.412875:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6f9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-mdtlov_UUID:3308:1863837562082176:0@lo:9:umount.0 00000100:00100000:0.0:1777502526.412886:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562082048:0@lo:103:ldlm_bl.0 00000100:00100000:0.0:1777502526.412897:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f595c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562081664:0@lo:103:ldlm_bl.0 00010000:00010000:0.0:1777502526.412936:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00080000:2.0:1777502526.412951:0:221612:0:(genops.c:1579:class_disconnect_exports()) OBD device 5 (ffff8dfd4f2e19c0) has no exports 00000020:00080000:2.0:1777502526.413201:0:221612:0:(genops.c:1579:class_disconnect_exports()) OBD device 5 (ffff8dfd4f2e19c0) has no exports 00000020:00000080:2.0:1777502526.435457:0:221612:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x4ad31463d9b9e512 00000020:00000080:2.0:1777502526.435499:0:221612:0:(genops.c:906:class_export_put()) final put ffff8dfc51710000/lustre-QMT0000_UUID 00000020:01000000:2.0:1777502526.435517:0:221612:0:(obd_config.c:2475:class_manual_cleanup()) Manual cleanup of lustre-QMT0000 (flags='') 00000020:00000080:0.0:1777502526.435541:0:194113:0:(genops.c:865:class_export_destroy()) destroying export ffff8dfc51710000/lustre-QMT0000_UUID for lustre-QMT0000 00000020:00000080:3.0:1777502526.435594:0:221612:0:(obd_config.c:1470:class_process_config()) processing cmd: cf004 00000020:00000080:3.0:1777502526.435604:0:221612:0:(obd_config.c:955:class_cleanup()) lustre-QMT0000: forcing exports to disconnect: 0/1 00000020:00080000:3.0:1777502526.435609:0:221612:0:(genops.c:1579:class_disconnect_exports()) OBD device 7 (ffff8dfd4f2e4d40) has no exports 00000020:00000080:3.0:1777502526.435955:0:221612:0:(genops.c:1489:class_disconnect()) disconnect: cookie 0x4ad31463d9b9e50b 00000020:00000080:3.0:1777502526.435961:0:221612:0:(genops.c:906:class_export_put()) final put ffff8dfd7f615000/lustre-MDT0000-osd_UUID 00000020:00000080:0.0:1777502526.435997:0:194113:0:(genops.c:865:class_export_destroy()) destroying export ffff8dfd7f615000/lustre-MDT0000-osd_UUID for lustre-MDT0000-osd 00000020:00000080:3.0:1777502526.436031:0:221612:0:(obd_config.c:1470:class_process_config()) processing cmd: cf002 00000020:00000080:3.0:1777502526.436039:0:221612:0:(obd_config.c:883:class_detach()) detach on obd lustre-QMT0000 (uuid lustre-QMT0000_UUID) 00000020:00000080:3.0:1777502526.436043:0:221612:0:(genops.c:906:class_export_put()) final put ffff8dfc429c7800/lustre-QMT0000_UUID 00000020:00000080:3.0:1777502526.436046:0:221612:0:(genops.c:865:class_export_destroy()) destroying export ffff8dfc429c7800/lustre-QMT0000_UUID for lustre-QMT0000 00000020:01000000:3.0:1777502526.436058:0:221612:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-QMT0000 (lustre-QMT0000_UUID) 00010000:00010000:3.0:1777502526.929748:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502527.953733:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502528.977738:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00100000:2.0:1777502529.115906:0:5817:0:(tgt_lastrcvd.c:568:tgt_cb_new_client()) lustre-MDT0000: committing for initial connect of 0775d2ab-558e-41f1-82c0-14244d133390 00000001:00080000:2.0:1777502529.115921:0:5817:0:(tgt_lastrcvd.c:988:tgt_cb_last_committed()) lustre-MDT0000: transno 8589949329 is committed 00010000:00010000:0.0:1777502530.001752:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1777502530.641807:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562082432:0@lo:400:kworker.0 00000100:00100000:3.0:1777502530.641849:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a058380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3312:1863837562082688:0@lo:400:kworker.0 00000100:00100000:0.0:1777502530.641872:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502530.641898:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502530.641935:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:3.0:1777502530.641942:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3312:1863837562082560:0@lo:400:kworker.0 00000100:00100000:3.0:1777502530.641973:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502530.642019:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562082432 00000100:00100000:3.0:1777502530.642027:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562082432 00000100:00100000:3.0:1777502530.642082:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 395 00000100:00100000:0.0:1777502530.642089:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562082688 00000100:00100000:3.0:1777502530.642095:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1d450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562082432:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502530.642096:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562082688 00000100:00100000:2.0:1777502530.642119:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562082560 00000100:00100000:2.0:1777502530.642124:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562082560 00000100:00100000:0.0:1777502530.642139:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16660 00000100:00100000:0.0:1777502530.642147:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a059880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3312:x1863837562082688:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502530.642153:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1d450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562082432:12345-0@lo:400:kworker.0 Request processed in 61us (278us total) trans 0 rc 0/0 00000020:00080000:0.0:1777502530.642159:0:205372:0:(tgt_handler.c:795:tgt_request_handle()) operation 400 on unconnected OST from 12345-0@lo 00000100:00100000:2.0:1777502530.642163:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16661 00000100:00100000:3.0:1777502530.642167:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 395 00000100:00100000:2.0:1777502530.642175:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a05a680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3312:x1863837562082560:12345-0@lo:400:kworker.0 00000020:00080000:2.0:1777502530.642182:0:205369:0:(tgt_handler.c:795:tgt_request_handle()) operation 400 on unconnected OST from 12345-0@lo 00000100:00100000:0.0:1777502530.642208:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a059880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3312:x1863837562082688:12345-0@lo:400:kworker.0 Request processed in 60us (310us total) trans 0 rc -107/-107 00000100:00100000:0.0:1777502530.642215:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16660 00000100:00100000:2.0:1777502530.642224:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a05a680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3312:x1863837562082560:12345-0@lo:400:kworker.0 Request processed in 47us (250us total) trans 0 rc -107/-107 00000100:00100000:2.0:1777502530.642230:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16661 00000100:00100000:3.0:1777502530.642240:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562082432:0@lo:400:kworker.0 00000100:00080000:2.0:1777502530.642273:0:3312:0:(recover.c:203:ptlrpc_request_handle_notconn()) import lustre-MDT0000-lwp-OST0000 of lustre-MDT0000_UUID@0@lo abruptly disconnected: reconnecting 00000100:00080000:2.0:1777502530.642278:0:3312:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1777502530.642282:0:3312:0:(import.c:181:ptlrpc_set_import_discon()) lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1777502530.649438:0:3312:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502530.649449:0:3312:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 7950 00000100:00080000:2.0:1777502530.649458:0:3312:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502530.649571:0:3312:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd6e6f9c00 x1863837562082816/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502530.649636:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3312:1863837562082560:0@lo:400:kworker.0 00000100:00100000:2.0:1777502530.649856:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6f9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562082816:0@lo:38:ptlrpcd_00_03.0 00000100:00100000:2.0:1777502530.649929:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502530.649991:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562082816 00000100:00100000:2.0:1777502530.649999:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562082816 00000100:00100000:2.0:1777502530.650038:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16662 00000100:00100000:2.0:1777502530.650047:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6f9f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562082816:12345-0@lo:38:ptlrpcd_00_03.0 00010000:02000000:2.0:1777502530.650064:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00080000:3.0:1777502530.652399:0:3312:0:(recover.c:203:ptlrpc_request_handle_notconn()) import lustre-MDT0000-lwp-OST0001 of lustre-MDT0000_UUID@0@lo abruptly disconnected: reconnecting 00000100:00080000:3.0:1777502530.652411:0:3312:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1777502530.652417:0:3312:0:(import.c:181:ptlrpc_set_import_discon()) lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1777502530.652423:0:3312:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502530.652427:0:3312:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 7955 00000100:00080000:3.0:1777502530.652434:0:3312:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502530.652458:0:3312:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a05a300 x1863837562082944/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502530.652484:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a058380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3312:1863837562082688:0@lo:400:kworker.0 00000100:00100000:3.0:1777502530.652564:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562082944:0@lo:38:ptlrpcd_00_03.0 00000100:00100000:3.0:1777502530.652636:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502530.652699:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562082944 00000100:00100000:3.0:1777502530.652704:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562082944 00000100:00100000:3.0:1777502530.652740:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16663 00000100:00100000:3.0:1777502530.652748:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a058380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562082944:12345-0@lo:38:ptlrpcd_00_03.0 00010000:02000000:3.0:1777502530.652762:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502530.652813:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a058380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562082944:12345-0@lo:38:ptlrpcd_00_03.0 Request processed in 63us (175us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502530.652821:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16663 00000100:00080000:3.0:1777502530.652866:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502530.652872:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502530.652877:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562082944:0@lo:38:ptlrpcd_00_03.0 00000001:00100000:0.0:1777502530.656966:0:6599:0:(tgt_lastrcvd.c:568:tgt_cb_new_client()) lustre-OST0000: committing for initial connect of lustre-MDT0000-mdtlov_UUID 00000020:00000080:0.0:1777502530.656971:0:6599:0:(genops.c:906:class_export_put()) final put ffff8dfd66bff000/lustre-MDT0000-mdtlov_UUID 00000020:00000080:0.0:1777502530.657001:0:194113:0:(genops.c:865:class_export_destroy()) destroying export ffff8dfd66bff000/lustre-MDT0000-mdtlov_UUID for lustre-OST0000 00000020:00000080:0.0:1777502530.657049:0:194113:0:(genops.c:1204:obd_zombie_import_free()) destroying import ffff8dfd66bfe000 for lustre-OST0000 00000100:00100000:2.0:1777502530.657870:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6f9f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562082816:12345-0@lo:38:ptlrpcd_00_03.0 Request processed in 7819us (7938us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502530.657883:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16662 00000100:00080000:0.0:1777502530.657898:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502530.657905:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502530.657910:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6f9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562082816:0@lo:38:ptlrpcd_00_03.0 00000001:00100000:3.0:1777502530.660008:0:7467:0:(tgt_lastrcvd.c:568:tgt_cb_new_client()) lustre-OST0001: committing for initial connect of lustre-MDT0000-mdtlov_UUID 00000020:00000080:3.0:1777502530.660015:0:7467:0:(genops.c:906:class_export_put()) final put ffff8dfd46534000/lustre-MDT0000-mdtlov_UUID 00000020:00000080:3.0:1777502530.660050:0:173604:0:(genops.c:865:class_export_destroy()) destroying export ffff8dfd46534000/lustre-MDT0000-mdtlov_UUID for lustre-OST0001 00000020:00000080:3.0:1777502530.660139:0:173604:0:(genops.c:1204:obd_zombie_import_free()) destroying import ffff8dfd46532800 for lustre-OST0001 00010000:00010000:3.0:1777502531.025757:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502532.049718:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502533.073760:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502534.097735:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502535.121743:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502535.761699:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502535.761704:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502535.761708:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502535.761716:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8657 00000100:00080000:3.0:1777502535.761721:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 5s 00000100:00100000:0.0:1777502535.761725:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562083072:0@lo:400:kworker.0 00000100:00080000:3.0:1777502535.761727:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502535.761746:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a058a80 x1863837562083200/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502535.761761:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502535.761763:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502535.761765:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502535.761768:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8657 00000100:00080000:3.0:1777502535.761771:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1777502535.761774:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502535.761784:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a05b800 x1863837562083328/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502535.761827:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502535.761862:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562083072 00000100:00100000:3.0:1777502535.761868:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562083072 00000100:00100000:3.0:1777502535.761912:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 396 00000100:00100000:3.0:1777502535.761921:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1f450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562083072:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502535.761972:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1f450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562083072:12345-0@lo:400:kworker.0 Request processed in 53us (153us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502535.761978:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 396 00000100:00100000:3.0:1777502535.762043:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562083200:0@lo:38:kworker.0 00000100:00100000:3.0:1777502535.762074:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502535.762097:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562083328:0@lo:38:kworker.0 00000100:00100000:3.0:1777502535.762116:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502535.762152:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562083200 00000100:00100000:3.0:1777502535.762155:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562083200 00000100:00100000:2.0:1777502535.762158:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562083328 00000100:00100000:0.0:1777502535.762159:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562083072:0@lo:400:kworker.0 00000100:00100000:2.0:1777502535.762165:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562083328 00000100:00100000:3.0:1777502535.762177:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16664 00000100:00100000:3.0:1777502535.762182:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a05bb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562083200:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502535.762195:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502535.762205:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16665 00000100:00100000:2.0:1777502535.762212:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a059180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562083328:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502535.762221:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502535.762258:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a059180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562083328:12345-0@lo:38:kworker.0 Request processed in 46us (143us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502535.762264:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16665 00000100:00080000:2.0:1777502535.762324:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502535.762330:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502535.762335:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562083328:0@lo:38:kworker.0 00000100:00100000:3.0:1777502535.765796:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a05bb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562083200:12345-0@lo:38:kworker.0 Request processed in 3611us (3720us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502535.765803:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16664 00000100:00080000:0.0:1777502535.765828:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502535.765833:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502535.765837:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562083200:0@lo:38:kworker.0 00010000:00010000:0.0:1777502536.145738:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502537.169737:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502538.193791:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502539.217720:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502540.241759:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:02000400:2.0:1777502540.881721:0:221612:0:(genops.c:1873:obd_exports_barrier()) lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 5. Is it stuck? 00000100:00080000:3.0:1777502540.882938:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502540.882944:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502540.882948:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502540.882956:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8663 00000100:00080000:3.0:1777502540.882961:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1777502540.882967:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502540.882973:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562083456:0@lo:400:kworker.0 00000100:00100000:3.0:1777502540.883004:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a058a80 x1863837562083584/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502540.883025:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502540.883029:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502540.883032:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502540.883047:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8663 00000100:00080000:3.0:1777502540.883050:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1777502540.883081:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502540.883082:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502540.883101:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a05b800 x1863837562083712/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502540.883162:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562083584:0@lo:38:kworker.0 00000100:00100000:0.0:1777502540.883219:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502540.883253:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562083712:0@lo:38:kworker.0 00000100:00100000:0.0:1777502540.883284:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502540.883327:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562083456 00000100:00100000:0.0:1777502540.883334:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562083456 00000100:00100000:0.0:1777502540.883400:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 397 00000100:00100000:0.0:1777502540.883412:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1cc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562083456:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502540.883463:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1cc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562083456:12345-0@lo:400:kworker.0 Request processed in 54us (382us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502540.883469:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 397 00000100:00100000:0.0:1777502540.883523:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562083584 00000100:00100000:0.0:1777502540.883527:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562083584 00000100:00100000:3.0:1777502540.883573:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562083712 00000100:00100000:0.0:1777502540.883598:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16666 00000100:00100000:0.0:1777502540.883605:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f595c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562083584:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502540.883643:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562083712 00010000:02000000:0.0:1777502540.883648:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502540.883689:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16667 00000100:00100000:3.0:1777502540.883697:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f594e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562083712:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502540.883708:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502540.883756:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f594e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562083712:12345-0@lo:38:kworker.0 Request processed in 57us (471us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502540.883764:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16667 00000100:00100000:3.0:1777502540.884066:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562083456:0@lo:400:kworker.0 00000100:00080000:3.0:1777502540.884173:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502540.884179:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502540.884184:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562083712:0@lo:38:kworker.0 00000020:02000400:2.0:1777502540.885416:0:221612:0:(genops.c:1840:print_export_data()) lustre-MDT0000: UNLINKED ffff8dfd66b1b800 0775d2ab-558e-41f1-82c0-14244d133390 192.168.202.27@tcp 1 (0 0 0) 1 0 0 0: 0000000000000000 8589949329 stale:0 00000100:00100000:0.0:1777502540.887572:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f595c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562083584:12345-0@lo:38:kworker.0 Request processed in 3964us (4351us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502540.887603:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16666 00000100:00080000:0.0:1777502540.887646:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502540.887653:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502540.887658:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562083584:0@lo:38:kworker.0 00010000:00010000:0.0:1777502541.265692:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502542.289729:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502543.313760:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502544.337727:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502545.361784:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502546.001780:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502546.001785:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1777502546.001800:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6faa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562083840:0@lo:400:kworker.0 00000100:00080000:2.0:1777502546.001825:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502546.001832:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8668 00000100:00080000:2.0:1777502546.001838:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1777502546.001857:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502546.001880:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502546.001884:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd6e6f9c00 x1863837562083968/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502546.001900:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502546.001903:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502546.001905:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502546.001909:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8668 00000100:00080000:2.0:1777502546.001912:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1777502546.001922:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502546.001934:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd509d3800 x1863837562084096/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502546.001999:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562083840 00000100:00100000:0.0:1777502546.002005:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562083840 00000100:00100000:2.0:1777502546.002022:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6f9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562083968:0@lo:38:kworker.0 00000100:00100000:0.0:1777502546.002055:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 398 00000100:00100000:0.0:1777502546.002067:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1e850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562083840:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502546.002097:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502546.002126:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1e850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562083840:12345-0@lo:400:kworker.0 Request processed in 60us (244us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502546.002136:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd509d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562084096:0@lo:38:kworker.0 00000100:00100000:0.0:1777502546.002137:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 398 00000100:00100000:2.0:1777502546.002170:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502546.002226:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562083968 00000100:00100000:2.0:1777502546.002230:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562083968 00000100:00100000:2.0:1777502546.002258:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562084096 00000100:00100000:2.0:1777502546.002261:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562084096 00000100:00100000:2.0:1777502546.002279:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16668 00000100:00100000:2.0:1777502546.002286:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d1f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562083968:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502546.002302:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502546.002541:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6faa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562083840:0@lo:400:kworker.0 00000100:00100000:0.0:1777502546.002844:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16669 00000100:00100000:0.0:1777502546.002850:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d0380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562084096:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502546.002858:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502546.002891:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d0380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562084096:12345-0@lo:38:kworker.0 Request processed in 41us (721us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502546.002946:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16669 00000100:00080000:0.0:1777502546.003019:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502546.003025:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502546.003029:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd509d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562084096:0@lo:38:kworker.0 00000100:00100000:2.0:1777502546.006977:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d1f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562083968:12345-0@lo:38:kworker.0 Request processed in 4687us (4878us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502546.007012:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16668 00000100:00080000:0.0:1777502546.007034:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502546.007042:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502546.007048:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6f9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562083968:0@lo:38:kworker.0 00010000:00010000:0.0:1777502546.385780:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502547.409892:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502548.433712:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502549.457780:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502550.481755:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502551.121729:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502551.121736:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502551.121739:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502551.121747:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8673 00000100:00080000:3.0:1777502551.121751:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1777502551.121757:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00080000:3.0:1777502551.121757:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502551.121778:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562084224:0@lo:400:kworker.0 00000100:00100000:3.0:1777502551.121779:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a05b100 x1863837562084352/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502551.121865:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502551.121868:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502551.121870:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502551.121874:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8673 00000100:00080000:3.0:1777502551.121876:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1777502551.121881:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:3.0:1777502551.121883:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502551.121895:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a05a300 x1863837562084480/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502551.121932:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562084224 00000100:00100000:3.0:1777502551.121938:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562084224 00000100:00100000:0.0:1777502551.121948:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562084352:0@lo:38:kworker.0 00000100:00100000:3.0:1777502551.121978:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 399 00000100:00100000:0.0:1777502551.121991:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502551.121991:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1f850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562084224:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502551.122030:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562084480:0@lo:38:kworker.0 00000100:00100000:3.0:1777502551.122056:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1f850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562084224:12345-0@lo:400:kworker.0 Request processed in 70us (177us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502551.122063:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 399 00000100:00100000:0.0:1777502551.122069:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502551.122120:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562084352 00000100:00100000:0.0:1777502551.122124:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562084352 00000100:00100000:3.0:1777502551.122130:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562084480 00000100:00100000:3.0:1777502551.122135:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562084480 00000100:00100000:0.0:1777502551.122161:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16670 00000100:00100000:3.0:1777502551.122173:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16671 00000100:00100000:3.0:1777502551.122179:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f594000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562084480:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502551.122192:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502551.123334:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562084224:0@lo:400:kworker.0 00000100:00100000:0.0:1777502551.123753:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f597480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562084352:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502551.123764:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502551.123811:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f597480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562084352:12345-0@lo:38:kworker.0 Request processed in 1643us (1826us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502551.123843:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16670 00000100:00080000:2.0:1777502551.123932:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502551.123937:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502551.123940:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562084352:0@lo:38:kworker.0 00000100:00100000:3.0:1777502551.126872:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f594000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562084480:12345-0@lo:38:kworker.0 Request processed in 4687us (4801us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502551.126885:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16671 00000100:00080000:0.0:1777502551.126911:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502551.126920:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502551.126925:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562084480:0@lo:38:kworker.0 00010000:00010000:3.0:1777502551.505717:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502552.529739:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502553.553755:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502554.577721:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502555.601720:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502556.241800:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502556.241805:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502556.241809:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502556.241815:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8678 00000100:00080000:2.0:1777502556.241819:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502556.241825:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502556.241830:0:3310:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1777502556.241844:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c6b480 x1863837562084736/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502556.241853:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd509d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562084608:0@lo:400:kworker.0 00000100:00080000:2.0:1777502556.241858:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502556.241861:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502556.241862:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502556.241866:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8678 00000100:00080000:2.0:1777502556.241868:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502556.241871:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502556.241881:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c68000 x1863837562084864/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502556.241951:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502556.242250:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562084736:0@lo:38:kworker.0 00000100:00100000:0.0:1777502556.242295:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502556.242321:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562084864:0@lo:38:kworker.0 00000100:00100000:0.0:1777502556.242350:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502556.242373:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562084736 00000100:00100000:3.0:1777502556.242381:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562084736 00000100:00100000:2.0:1777502556.242484:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562084864 00000100:00100000:2.0:1777502556.242489:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562084864 00000100:00100000:2.0:1777502556.242526:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16672 00000100:00100000:2.0:1777502556.242565:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f595180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562084864:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502556.242612:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502556.242664:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f595180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562084864:12345-0@lo:38:kworker.0 Request processed in 98us (312us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502556.242672:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16672 00000100:00100000:2.0:1777502556.242739:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562084608 00000100:00100000:2.0:1777502556.242744:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562084608 00000100:00100000:2.0:1777502556.242780:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 400 00000100:00100000:2.0:1777502556.242787:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1c050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562084608:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502556.242836:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1c050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562084608:12345-0@lo:400:kworker.0 Request processed in 49us (886us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502556.242843:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 400 00000100:00080000:0.0:1777502556.242856:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502556.242864:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502556.242869:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562084864:0@lo:38:kworker.0 00000100:00100000:2.0:1777502556.242920:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd509d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562084608:0@lo:400:kworker.0 00000100:00100000:3.0:1777502556.243181:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16673 00000100:00100000:3.0:1777502556.243188:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f596a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562084736:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502556.243198:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502556.243237:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f596a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562084736:12345-0@lo:38:kworker.0 Request processed in 48us (942us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502556.243245:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16673 00000100:00080000:0.0:1777502556.243306:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502556.243312:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502556.243316:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562084736:0@lo:38:kworker.0 00010000:00010000:0.0:1777502556.625774:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:02000400:2.0:1777502557.265661:0:221612:0:(genops.c:1873:obd_exports_barrier()) lustre-MDT0000 is waiting for obd_unlinked_exports more than 16 seconds. The obd refcount = 5. Is it stuck? 00000020:02000400:2.0:1777502557.269033:0:221612:0:(genops.c:1840:print_export_data()) lustre-MDT0000: UNLINKED ffff8dfd66b1b800 0775d2ab-558e-41f1-82c0-14244d133390 192.168.202.27@tcp 1 (0 0 0) 1 0 0 0: 0000000000000000 8589949329 stale:0 00010000:00010000:0.0:1777502557.649762:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502558.673746:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502559.697711:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502560.721738:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502561.361699:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502561.361705:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502561.361708:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502561.361715:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8683 00000100:00080000:2.0:1777502561.361720:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502561.361726:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502561.361747:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c6b800 x1863837562085120/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502561.361750:0:3309:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:2.0:1777502561.361767:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1777502561.361774:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562084992:0@lo:400:kworker.0 00000100:00080000:2.0:1777502561.361801:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502561.361803:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502561.361808:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8683 00000100:00080000:2.0:1777502561.361811:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502561.361815:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502561.361830:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c68700 x1863837562085248/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502561.361842:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502561.361918:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562085120:0@lo:38:kworker.0 00000100:00100000:0.0:1777502561.361946:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562084992 00000100:00100000:0.0:1777502561.361953:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562084992 00000100:00100000:2.0:1777502561.361958:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502561.361983:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562085248:0@lo:38:kworker.0 00000100:00100000:0.0:1777502561.362004:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 401 00000100:00100000:2.0:1777502561.362005:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502561.362039:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1ec50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562084992:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502561.362062:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562085120 00000100:00100000:3.0:1777502561.362067:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562085120 00000100:00100000:3.0:1777502561.362095:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562085248 00000100:00100000:3.0:1777502561.362098:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562085248 00000100:00100000:0.0:1777502561.362111:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1ec50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562084992:12345-0@lo:400:kworker.0 Request processed in 76us (270us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502561.362118:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16674 00000100:00100000:0.0:1777502561.362121:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 401 00000100:00100000:3.0:1777502561.362126:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c69500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562085120:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502561.362142:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502561.362169:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16675 00000100:00100000:0.0:1777502561.362176:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c69880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562085248:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502561.362187:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502561.362229:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c69880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562085248:12345-0@lo:38:kworker.0 Request processed in 52us (224us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502561.362237:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16675 00000100:00080000:2.0:1777502561.362297:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502561.362313:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502561.362318:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562085248:0@lo:38:kworker.0 00000100:00100000:0.0:1777502561.362559:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562084992:0@lo:400:kworker.0 00000100:00100000:3.0:1777502561.366453:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c69500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562085120:12345-0@lo:38:kworker.0 Request processed in 4322us (4489us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502561.366466:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16674 00000100:00080000:0.0:1777502561.366495:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502561.366502:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502561.366506:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562085120:0@lo:38:kworker.0 00010000:00010000:0.0:1777502561.745749:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502562.769829:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502563.793727:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502564.817762:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502565.841724:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502566.481777:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502566.481784:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502566.481788:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502566.481797:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8688 00000100:00080000:0.0:1777502566.481802:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502566.481810:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502566.481815:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f595f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562085376:0@lo:400:kworker.0 00000100:00100000:0.0:1777502566.481835:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f596d80 x1863837562085504/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502566.481903:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502566.481908:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1777502566.481911:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:0.0:1777502566.481912:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502566.481918:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8688 00000100:00080000:0.0:1777502566.481921:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502566.481924:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502566.481938:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f594380 x1863837562085632/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502566.481991:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f596d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562085504:0@lo:38:kworker.0 00000100:00100000:2.0:1777502566.482001:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562085376 00000100:00100000:2.0:1777502566.482008:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562085376 00000100:00100000:0.0:1777502566.482037:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502566.482059:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 402 00000100:00100000:2.0:1777502566.482070:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1fc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562085376:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502566.482072:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f594380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562085632:0@lo:38:kworker.0 00000100:00100000:0.0:1777502566.482104:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502566.482136:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1fc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562085376:12345-0@lo:400:kworker.0 Request processed in 67us (227us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502566.482143:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 402 00000100:00100000:0.0:1777502566.482158:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562085504 00000100:00100000:3.0:1777502566.482159:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562085632 00000100:00100000:0.0:1777502566.482163:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562085504 00000100:00100000:3.0:1777502566.482165:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562085632 00000100:00100000:3.0:1777502566.482198:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16676 00000100:00100000:2.0:1777502566.482203:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f595f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562085376:0@lo:400:kworker.0 00000100:00100000:3.0:1777502566.482206:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6faa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562085632:12345-0@lo:38:kworker.0 00000100:00100000:0.0:1777502566.482209:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16677 00000100:00100000:0.0:1777502566.482218:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6f9c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562085504:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502566.482237:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:3.0:1777502566.482237:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502566.482287:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6faa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562085632:12345-0@lo:38:kworker.0 Request processed in 82us (182us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502566.482292:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6f9c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562085504:12345-0@lo:38:kworker.0 Request processed in 73us (254us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502566.482296:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16676 00000100:00100000:0.0:1777502566.482301:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16677 00000100:00080000:0.0:1777502566.482368:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502566.482373:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502566.482376:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f596d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562085504:0@lo:38:kworker.0 00000100:00080000:0.0:1777502566.482389:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502566.482397:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502566.482401:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f594380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562085632:0@lo:38:kworker.0 00010000:00010000:3.0:1777502566.865717:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502567.889787:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502568.913715:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502569.937814:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502570.961755:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502571.601768:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502571.601774:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502571.601779:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502571.601788:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8693 00000100:00080000:3.0:1777502571.601793:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502571.601799:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502571.601823:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a05b100 x1863837562085888/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502571.601843:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502571.601846:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502571.601849:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502571.601855:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8693 00000100:00080000:3.0:1777502571.601858:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502571.601863:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502571.601879:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a05b800 x1863837562086016/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502571.601907:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1777502571.601925:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562085760:0@lo:400:kworker.0 00000100:00100000:3.0:1777502571.602063:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502571.602148:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562085760 00000100:00100000:3.0:1777502571.602155:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562085760 00000100:00100000:0.0:1777502571.602177:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562085888:0@lo:38:kworker.0 00000100:00100000:3.0:1777502571.602204:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 403 00000100:00100000:0.0:1777502571.602214:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502571.602218:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1d850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562085760:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502571.602244:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562086016:0@lo:38:kworker.0 00000100:00100000:3.0:1777502571.602296:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1d850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562085760:12345-0@lo:400:kworker.0 Request processed in 82us (233us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502571.602306:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 403 00000100:00100000:0.0:1777502571.602331:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502571.602397:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562085888 00000100:00100000:0.0:1777502571.602408:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562085888 00000100:00100000:3.0:1777502571.602410:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562086016 00000100:00100000:3.0:1777502571.602416:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562086016 00000100:00100000:0.0:1777502571.602442:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16678 00000100:00100000:0.0:1777502571.602449:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef2d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562085888:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502571.602464:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502571.602487:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16679 00000100:00100000:3.0:1777502571.602495:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef0e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562086016:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502571.602506:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502571.602519:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef2d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562085888:12345-0@lo:38:kworker.0 Request processed in 68us (304us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502571.602528:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16678 00000100:00100000:2.0:1777502571.602548:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562085760:0@lo:400:kworker.0 00000100:00100000:3.0:1777502571.602557:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef0e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562086016:12345-0@lo:38:kworker.0 Request processed in 60us (226us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502571.602566:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16679 00000100:00080000:0.0:1777502571.602609:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502571.602615:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502571.602619:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562086016:0@lo:38:kworker.0 00000100:00080000:0.0:1777502571.602643:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502571.602646:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502571.602650:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562085888:0@lo:38:kworker.0 00010000:00010000:3.0:1777502571.985829:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502573.009730:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502574.033728:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502575.057749:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502576.081710:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502576.721746:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502576.721789:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502576.721794:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502576.721803:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8698 00000100:00080000:0.0:1777502576.721808:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502576.721815:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502576.721840:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4cef2a00 x1863837562086272/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502576.721860:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502576.721864:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502576.721866:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502576.721871:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8698 00000100:00080000:0.0:1777502576.721874:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502576.721878:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502576.721905:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4cef1500 x1863837562086400/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502576.721938:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4cef3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562086144:0@lo:400:kworker.0 00000100:00100000:0.0:1777502576.722007:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502576.722100:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4cef2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562086272:0@lo:38:kworker.0 00000100:00100000:2.0:1777502576.722114:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562086144 00000100:00100000:2.0:1777502576.722121:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562086144 00000100:00100000:0.0:1777502576.722153:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502576.722164:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 404 00000100:00100000:2.0:1777502576.722173:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1e050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562086144:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502576.722184:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4cef1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562086400:0@lo:38:kworker.0 00000100:00100000:0.0:1777502576.722237:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502576.722289:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1e050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562086144:12345-0@lo:400:kworker.0 Request processed in 82us (247us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502576.722292:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562086272 00000100:00100000:0.0:1777502576.722297:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562086272 00000100:00100000:2.0:1777502576.722302:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 404 00000100:00100000:0.0:1777502576.722324:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562086400 00000100:00100000:0.0:1777502576.722328:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562086400 00000100:00100000:0.0:1777502576.722362:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16680 00000100:00100000:0.0:1777502576.722370:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef0700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562086272:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502576.722385:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502576.722429:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef0700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562086272:12345-0@lo:38:kworker.0 Request processed in 59us (278us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502576.722437:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16680 00000100:00100000:0.0:1777502576.722458:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16681 00000100:00100000:0.0:1777502576.722465:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef1180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562086400:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502576.722473:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502576.722507:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef1180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562086400:12345-0@lo:38:kworker.0 Request processed in 41us (272us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502576.722515:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16681 00000100:00100000:0.0:1777502576.722604:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4cef3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562086144:0@lo:400:kworker.0 00000100:00080000:0.0:1777502576.722646:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502576.722652:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502576.722657:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4cef2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562086272:0@lo:38:kworker.0 00000100:00080000:0.0:1777502576.722673:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502576.722678:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502576.722682:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4cef1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562086400:0@lo:38:kworker.0 00010000:00010000:0.0:1777502577.105731:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502578.129752:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502579.153728:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502580.177731:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502581.201723:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502581.841764:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502581.841770:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502581.841775:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502581.841783:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8703 00000100:00080000:2.0:1777502581.841788:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502581.841795:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502581.841820:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c6b800 x1863837562086656/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502581.841838:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502581.841842:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502581.841844:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502581.841849:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8703 00000100:00080000:2.0:1777502581.841852:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502581.841857:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502581.841873:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c68380 x1863837562086784/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502581.841950:0:3309:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1777502581.841968:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562086528:0@lo:400:kworker.0 00000100:00100000:2.0:1777502581.842040:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502581.842105:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562086656:0@lo:38:kworker.0 00000100:00100000:2.0:1777502581.842120:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562086528 00000100:00100000:2.0:1777502581.842128:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562086528 00000100:00100000:0.0:1777502581.842151:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502581.842193:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 405 00000100:00100000:0.0:1777502581.842195:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562086784:0@lo:38:kworker.0 00000100:00100000:2.0:1777502581.842206:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1d050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562086528:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502581.842265:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1d050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562086528:12345-0@lo:400:kworker.0 Request processed in 63us (226us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502581.842276:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 405 00000100:00100000:0.0:1777502581.842277:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502581.842331:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562086656 00000100:00100000:0.0:1777502581.842336:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562086656 00000100:00100000:2.0:1777502581.842337:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562086528:0@lo:400:kworker.0 00000100:00100000:0.0:1777502581.842364:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562086784 00000100:00100000:0.0:1777502581.842368:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562086784 00000100:00100000:0.0:1777502581.842407:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16682 00000100:00100000:0.0:1777502581.842414:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef1500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562086656:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502581.842431:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502581.842628:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16683 00000100:00100000:2.0:1777502581.842635:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef2a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562086784:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502581.842646:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502581.842689:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef2a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562086784:12345-0@lo:38:kworker.0 Request processed in 52us (414us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502581.842696:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16683 00000100:00080000:2.0:1777502581.842920:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502581.842926:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502581.842931:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562086784:0@lo:38:kworker.0 00000100:00100000:0.0:1777502581.846779:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef1500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562086656:12345-0@lo:38:kworker.0 Request processed in 4360us (4626us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502581.846792:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16682 00000100:00080000:3.0:1777502581.846889:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502581.846897:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502581.846903:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562086656:0@lo:38:kworker.0 00010000:00010000:0.0:1777502582.225747:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502583.249752:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502584.273747:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502585.297750:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502586.321732:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502586.961717:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1777502586.961743:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562086912:0@lo:400:kworker.0 00000100:00080000:3.0:1777502586.961779:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502586.961784:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502586.961790:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8709 00000100:00080000:3.0:1777502586.961795:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502586.961805:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502586.961806:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502586.961829:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a05b100 x1863837562087040/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502586.961846:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502586.961849:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502586.961851:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502586.961855:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8709 00000100:00080000:3.0:1777502586.961857:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502586.961861:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502586.961872:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a05b800 x1863837562087168/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502586.961908:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562086912 00000100:00100000:3.0:1777502586.961914:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562086912 00000100:00100000:3.0:1777502586.961960:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 406 00000100:00100000:3.0:1777502586.961970:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1e450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562086912:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502586.962029:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1e450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562086912:12345-0@lo:400:kworker.0 Request processed in 60us (223us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502586.962038:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 406 00000100:00100000:3.0:1777502586.962084:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562087040:0@lo:38:kworker.0 00000100:00100000:3.0:1777502586.962133:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502586.962160:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562087168:0@lo:38:kworker.0 00000100:00100000:3.0:1777502586.962214:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502586.962225:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562087040 00000100:00100000:0.0:1777502586.962231:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562087040 00000100:00100000:2.0:1777502586.962243:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562086912:0@lo:400:kworker.0 00000100:00100000:0.0:1777502586.962265:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562087168 00000100:00100000:0.0:1777502586.962270:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562087168 00000100:00100000:0.0:1777502586.962298:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16684 00000100:00100000:0.0:1777502586.962305:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a05a300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562087040:12345-0@lo:38:kworker.0 00000100:00100000:2.0:1777502586.962306:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16685 00010000:02000000:0.0:1777502586.962322:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502586.962335:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfcc2532300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562087168:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502586.962397:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502586.962403:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a05a300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562087040:12345-0@lo:38:kworker.0 Request processed in 95us (269us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502586.962411:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16684 00000100:00080000:0.0:1777502586.962458:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502586.962465:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502586.962470:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562087040:0@lo:38:kworker.0 00000100:00100000:3.0:1777502586.962615:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfcc2532300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562087168:12345-0@lo:38:kworker.0 Request processed in 279us (399us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502586.962625:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16685 00000100:00080000:3.0:1777502586.962681:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502586.962687:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502586.962692:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562087168:0@lo:38:kworker.0 00010000:00010000:3.0:1777502587.345763:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502588.369731:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502589.393723:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:02000400:2.0:1777502589.521665:0:221612:0:(genops.c:1873:obd_exports_barrier()) lustre-MDT0000 is waiting for obd_unlinked_exports more than 32 seconds. The obd refcount = 5. Is it stuck? 00000020:02000400:2.0:1777502589.524869:0:221612:0:(genops.c:1840:print_export_data()) lustre-MDT0000: UNLINKED ffff8dfd66b1b800 0775d2ab-558e-41f1-82c0-14244d133390 192.168.202.27@tcp 1 (0 0 0) 1 0 0 0: 0000000000000000 8589949329 stale:0 00010000:00010000:0.0:1777502590.417743:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502591.441739:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502592.081729:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1777502592.081740:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00080000:0.0:1777502592.081747:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502592.081753:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1777502592.081763:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4cef3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562087296:0@lo:400:kworker.0 00000100:00080000:0.0:1777502592.081767:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8714 00000100:00080000:0.0:1777502592.081773:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502592.081783:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502592.081807:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4cef1880 x1863837562087424/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502592.081826:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502592.081830:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502592.081833:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502592.081838:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8714 00000100:00080000:0.0:1777502592.081841:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502592.081845:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502592.081861:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4cef2680 x1863837562087552/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502592.081908:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502592.082006:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562087296 00000100:00100000:2.0:1777502592.082013:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562087296 00000100:00100000:2.0:1777502592.082081:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 407 00000100:00100000:2.0:1777502592.082093:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1c450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562087296:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502592.082116:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4cef1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562087424:0@lo:38:kworker.0 00000100:00100000:3.0:1777502592.082150:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502592.082174:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4cef2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562087552:0@lo:38:kworker.0 00000100:00100000:3.0:1777502592.082209:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502592.082227:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1c450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562087296:12345-0@lo:400:kworker.0 Request processed in 136us (319us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502592.082237:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 407 00000100:00100000:2.0:1777502592.082295:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562087424 00000100:00100000:3.0:1777502592.082299:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562087552 00000100:00100000:2.0:1777502592.082300:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562087424 00000100:00100000:3.0:1777502592.082307:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562087552 00000100:00100000:2.0:1777502592.082334:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16686 00000100:00100000:2.0:1777502592.082340:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfcc2531180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562087424:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502592.082342:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16687 00000100:00100000:3.0:1777502592.082354:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e4f6680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562087552:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502592.082359:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:3.0:1777502592.082366:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502592.082412:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfcc2531180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562087424:12345-0@lo:38:kworker.0 Request processed in 69us (260us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502592.082416:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e4f6680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562087552:12345-0@lo:38:kworker.0 Request processed in 62us (205us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502592.082427:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16686 00000100:00100000:3.0:1777502592.082434:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16687 00000100:00100000:2.0:1777502592.082478:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4cef3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562087296:0@lo:400:kworker.0 00000100:00080000:0.0:1777502592.082524:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502592.082530:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502592.082536:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4cef1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562087424:0@lo:38:kworker.0 00000100:00080000:0.0:1777502592.082552:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502592.082557:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502592.082562:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4cef2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562087552:0@lo:38:kworker.0 00010000:00010000:3.0:1777502592.465721:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502593.489732:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502594.513723:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502595.537716:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502596.561744:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502597.201739:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502597.201747:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502597.201761:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502597.201770:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8719 00000100:00080000:2.0:1777502597.201776:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502597.201784:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502597.201821:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c68700 x1863837562087808/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502597.201842:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502597.201847:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502597.201849:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502597.201854:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8719 00000100:00080000:2.0:1777502597.201858:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502597.201862:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502597.201877:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c6b480 x1863837562087936/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502597.201909:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562087680:0@lo:400:kworker.0 00000100:00100000:2.0:1777502597.202041:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502597.202119:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562087808:0@lo:38:kworker.0 00000100:00100000:2.0:1777502597.202129:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562087680 00000100:00100000:2.0:1777502597.202137:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562087680 00000100:00100000:0.0:1777502597.202153:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502597.202180:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562087936:0@lo:38:kworker.0 00000100:00100000:2.0:1777502597.202189:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 408 00000100:00100000:2.0:1777502597.202201:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee26050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562087680:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502597.202213:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502597.202261:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562087808 00000100:00100000:2.0:1777502597.202262:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee26050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562087680:12345-0@lo:400:kworker.0 Request processed in 63us (221us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502597.202265:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562087808 00000100:00100000:2.0:1777502597.202274:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 408 00000100:00100000:2.0:1777502597.202342:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562087680:0@lo:400:kworker.0 00000100:00100000:0.0:1777502597.202345:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562087936 00000100:00100000:0.0:1777502597.202348:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562087936 00000100:00100000:0.0:1777502597.202373:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16688 00000100:00100000:0.0:1777502597.202378:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef1880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562087936:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502597.202391:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502597.202439:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef1880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562087936:12345-0@lo:38:kworker.0 Request processed in 59us (227us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502597.202447:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16688 00000100:00100000:0.0:1777502597.202496:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16689 00000100:00100000:0.0:1777502597.202502:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef2680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562087808:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502597.202512:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502597.202550:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef2680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562087808:12345-0@lo:38:kworker.0 Request processed in 46us (396us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502597.202557:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16689 00000100:00080000:0.0:1777502597.202608:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502597.202614:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502597.202618:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562087808:0@lo:38:kworker.0 00000100:00080000:0.0:1777502597.202631:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502597.202636:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502597.202640:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562087936:0@lo:38:kworker.0 00010000:00010000:0.0:1777502597.585743:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502598.609719:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502599.633773:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502600.657756:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502601.681721:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502602.321688:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502602.321704:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502602.321707:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502602.321714:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8724 00000100:00080000:2.0:1777502602.321717:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502602.321724:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502602.321780:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c68000 x1863837562088192/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502602.321799:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502602.321802:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502602.321804:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502602.321818:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8724 00000100:00080000:2.0:1777502602.321820:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502602.321824:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502602.321838:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c6b480 x1863837562088320/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502602.321865:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562088064:0@lo:400:kworker.0 00000100:00100000:2.0:1777502602.321935:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502602.322024:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562088064 00000100:00100000:2.0:1777502602.322031:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562088064 00000100:00100000:0.0:1777502602.322059:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562088192:0@lo:38:kworker.0 00000100:00100000:2.0:1777502602.322092:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 409 00000100:00100000:2.0:1777502602.322105:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee25050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562088064:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502602.322116:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502602.322143:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562088320:0@lo:38:kworker.0 00000100:00100000:2.0:1777502602.322178:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee25050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562088064:12345-0@lo:400:kworker.0 Request processed in 76us (242us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502602.322192:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 409 00000100:00100000:0.0:1777502602.322237:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502602.322257:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562088064:0@lo:400:kworker.0 00000100:00100000:0.0:1777502602.322305:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562088192 00000100:00100000:0.0:1777502602.322309:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562088192 00000100:00100000:0.0:1777502602.322344:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562088320 00000100:00100000:0.0:1777502602.322347:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562088320 00000100:00100000:0.0:1777502602.322380:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16690 00000100:00100000:0.0:1777502602.322386:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef1c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562088192:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502602.322391:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16691 00000100:00100000:3.0:1777502602.322398:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef0000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562088320:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502602.322406:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:3.0:1777502602.322409:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502602.322446:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef1c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562088192:12345-0@lo:38:kworker.0 Request processed in 58us (330us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502602.322458:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16690 00000100:00100000:3.0:1777502602.322458:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef0000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562088320:12345-0@lo:38:kworker.0 Request processed in 57us (266us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502602.322469:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16691 00000100:00080000:0.0:1777502602.322523:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502602.322528:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502602.322532:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562088320:0@lo:38:kworker.0 00000100:00080000:0.0:1777502602.322564:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502602.322569:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502602.322574:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562088192:0@lo:38:kworker.0 00010000:00010000:3.0:1777502602.705737:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502603.729761:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502604.753729:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502605.777723:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502606.801810:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502607.441749:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502607.441755:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502607.441760:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1777502607.441790:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4cef3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562088448:0@lo:400:kworker.0 00000100:00080000:0.0:1777502607.441820:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8729 00000100:00080000:0.0:1777502607.441823:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502607.441833:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502607.441857:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4cef0a80 x1863837562088576/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502607.441878:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502607.441883:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502607.441885:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502607.441889:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8729 00000100:00080000:0.0:1777502607.441892:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502607.441897:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502607.441904:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502607.441931:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4cef3b80 x1863837562088704/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502607.441978:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4cef0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562088576:0@lo:38:kworker.0 00000100:00100000:0.0:1777502607.442049:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502607.442056:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562088448 00000100:00100000:2.0:1777502607.442064:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562088448 00000100:00100000:2.0:1777502607.442146:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 410 00000100:00100000:2.0:1777502607.442159:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee26450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562088448:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502607.442226:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee26450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562088448:12345-0@lo:400:kworker.0 Request processed in 71us (325us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502607.442236:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 410 00000100:00100000:2.0:1777502607.442305:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562088576 00000100:00100000:2.0:1777502607.442310:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562088576 00000100:00100000:2.0:1777502607.442724:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16692 00000100:00100000:2.0:1777502607.442732:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef3480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562088576:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502607.442749:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502607.442810:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef3480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562088576:12345-0@lo:38:kworker.0 Request processed in 75us (765us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502607.442819:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16692 00000100:00100000:3.0:1777502607.442901:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4cef3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562088448:0@lo:400:kworker.0 00000100:00080000:0.0:1777502607.443198:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502607.443208:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502607.443215:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4cef0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562088576:0@lo:38:kworker.0 00000100:00100000:0.0:1777502607.443226:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4cef3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562088704:0@lo:38:kworker.0 00000100:00100000:0.0:1777502607.443271:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502607.443338:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562088704 00000100:00100000:0.0:1777502607.443342:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562088704 00000100:00100000:0.0:1777502607.443370:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16693 00000100:00100000:0.0:1777502607.443376:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef3800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562088704:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502607.443383:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502607.443438:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef3800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562088704:12345-0@lo:38:kworker.0 Request processed in 60us (173us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502607.443447:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16693 00000100:00080000:2.0:1777502607.443477:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502607.443486:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502607.443491:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4cef3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562088704:0@lo:38:kworker.0 00010000:00010000:3.0:1777502607.825723:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502608.849739:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502609.873760:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502610.897761:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502611.921725:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502612.561735:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502612.561740:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502612.561744:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502612.561751:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8734 00000100:00080000:0.0:1777502612.561755:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502612.561760:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502612.561780:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4cef3b80 x1863837562088960/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502612.561800:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502612.561804:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502612.561807:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502612.561811:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8734 00000100:00080000:0.0:1777502612.561813:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502612.561825:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502612.561847:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a80df80 x1863837562089088/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502612.561886:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4cef0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562088832:0@lo:400:kworker.0 00000100:00100000:0.0:1777502612.561973:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502612.562096:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562088832 00000100:00100000:0.0:1777502612.562103:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562088832 00000100:00100000:0.0:1777502612.562157:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 411 00000100:00100000:2.0:1777502612.562157:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4cef3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562088960:0@lo:38:kworker.0 00000100:00100000:0.0:1777502612.562172:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee24450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562088832:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502612.562213:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502612.562239:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a80df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562089088:0@lo:38:kworker.0 00000100:00100000:0.0:1777502612.562260:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee24450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562088832:12345-0@lo:400:kworker.0 Request processed in 90us (289us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502612.562267:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 411 00000100:00100000:0.0:1777502612.562323:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562088960 00000100:00100000:0.0:1777502612.562327:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562088960 00000100:00100000:2.0:1777502612.562367:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502612.562371:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16694 00000100:00100000:0.0:1777502612.562378:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c68380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562088960:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502612.562393:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502612.562433:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562089088 00000100:00100000:0.0:1777502612.562440:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c68380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562088960:12345-0@lo:38:kworker.0 Request processed in 60us (226us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502612.562447:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16694 00000100:00100000:2.0:1777502612.562458:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562089088 00000100:00100000:0.0:1777502612.562491:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16695 00000100:00100000:0.0:1777502612.562496:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c68700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562089088:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502612.562503:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502612.562531:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c68700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562089088:12345-0@lo:38:kworker.0 Request processed in 34us (169us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502612.562537:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16695 00000100:00080000:2.0:1777502612.562548:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502612.562553:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502612.562558:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4cef3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562088960:0@lo:38:kworker.0 00000100:00100000:0.0:1777502612.562592:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4cef0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562088832:0@lo:400:kworker.0 00000100:00080000:2.0:1777502612.562603:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502612.562609:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502612.562614:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a80df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562089088:0@lo:38:kworker.0 00010000:00010000:3.0:1777502612.945736:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502613.969760:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502614.993797:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502616.017747:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502617.041739:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502617.681842:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502617.681847:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502617.681851:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502617.681857:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8739 00000100:00080000:2.0:1777502617.681861:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502617.681878:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502617.681903:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c68000 x1863837562089344/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502617.681918:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502617.681921:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502617.681923:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502617.681926:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8739 00000100:00080000:2.0:1777502617.681928:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502617.681930:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502617.681943:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c6b480 x1863837562089472/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502617.681969:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562089216:0@lo:400:kworker.0 00000100:00100000:2.0:1777502617.682060:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502617.682144:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562089216 00000100:00100000:3.0:1777502617.682153:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562089216 00000100:00100000:3.0:1777502617.682204:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 412 00000100:00100000:3.0:1777502617.682214:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee24850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562089216:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502617.682273:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee24850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562089216:12345-0@lo:400:kworker.0 Request processed in 60us (212us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502617.682274:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562089344:0@lo:38:kworker.0 00000100:00100000:3.0:1777502617.682287:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 412 00000100:00100000:2.0:1777502617.682373:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502617.682391:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562089216:0@lo:400:kworker.0 00000100:00100000:3.0:1777502617.682475:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562089472:0@lo:38:kworker.0 00000100:00100000:3.0:1777502617.682508:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502617.682553:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562089344 00000100:00100000:3.0:1777502617.682556:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562089472 00000100:00100000:2.0:1777502617.682558:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562089344 00000100:00100000:3.0:1777502617.682561:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562089472 00000100:00100000:2.0:1777502617.682620:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16696 00000100:00100000:3.0:1777502617.682629:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16697 00000100:00100000:2.0:1777502617.682630:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a05b800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562089344:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502617.682637:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e4f7800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562089472:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502617.682648:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:3.0:1777502617.682649:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502617.682697:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a05b800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562089344:12345-0@lo:38:kworker.0 Request processed in 69us (324us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502617.682705:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16696 00000100:00080000:2.0:1777502617.682746:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502617.682752:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502617.682756:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562089344:0@lo:38:kworker.0 00000100:00100000:3.0:1777502617.686944:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e4f7800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562089472:12345-0@lo:38:kworker.0 Request processed in 4304us (4435us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502617.686953:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16697 00000100:00080000:0.0:1777502617.686993:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502617.687000:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502617.687008:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562089472:0@lo:38:kworker.0 00010000:00010000:0.0:1777502618.065729:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502619.089856:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502620.113753:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502621.137817:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502622.161814:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502622.801726:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502622.801731:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502622.801735:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502622.801744:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8744 00000100:00080000:2.0:1777502622.801754:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502622.801761:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502622.801794:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a058a80 x1863837562089728/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502622.801816:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502622.801820:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502622.801822:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502622.801843:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8744 00000100:00080000:2.0:1777502622.801846:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502622.801852:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502622.801873:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f594380 x1863837562089856/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502622.801906:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562089600:0@lo:400:kworker.0 00000100:00100000:2.0:1777502622.801969:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502622.802045:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562089728:0@lo:38:kworker.0 00000100:00100000:2.0:1777502622.802072:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502622.802110:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f594380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562089856:0@lo:38:kworker.0 00000100:00100000:2.0:1777502622.802140:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502622.802183:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562089728 00000100:00100000:3.0:1777502622.802191:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562089728 00000100:00100000:3.0:1777502622.802229:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562089856 00000100:00100000:3.0:1777502622.802233:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562089856 00000100:00100000:3.0:1777502622.802263:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16698 00000100:00100000:3.0:1777502622.802272:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f596d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562089728:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502622.802290:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502622.802344:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f596d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562089728:12345-0@lo:38:kworker.0 Request processed in 70us (269us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502622.802353:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16698 00000100:00100000:2.0:1777502622.802360:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562089600 00000100:00100000:2.0:1777502622.802365:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562089600 00000100:00100000:0.0:1777502622.802373:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16699 00000100:00100000:2.0:1777502622.802408:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 413 00000100:00100000:2.0:1777502622.802419:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee27050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562089600:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502622.802435:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f595f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562089856:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502622.802451:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00080000:3.0:1777502622.802466:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502622.802473:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502622.802479:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562089728:0@lo:38:kworker.0 00000100:00100000:2.0:1777502622.802482:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee27050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562089600:12345-0@lo:400:kworker.0 Request processed in 63us (511us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502622.802493:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 413 00000100:00100000:3.0:1777502622.802557:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562089600:0@lo:400:kworker.0 00000100:00100000:0.0:1777502622.802716:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f595f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562089856:12345-0@lo:38:kworker.0 Request processed in 282us (576us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502622.802722:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16699 00000100:00080000:0.0:1777502622.802756:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502622.802762:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502622.802776:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f594380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562089856:0@lo:38:kworker.0 00010000:00010000:3.0:1777502623.185787:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502624.209741:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502625.233775:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502626.257740:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502627.281699:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502627.921755:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502627.921760:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502627.921764:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502627.921772:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8750 00000100:00080000:2.0:1777502627.921776:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502627.921782:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502627.921804:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd509d3800 x1863837562090112/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502627.921830:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502627.921833:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502627.921835:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502627.921839:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8750 00000100:00080000:2.0:1777502627.921841:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502627.921844:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502627.921859:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc44799880 x1863837562090240/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502627.921907:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:2.0:1777502627.921923:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f594380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562089984:0@lo:400:kworker.0 00000100:00100000:2.0:1777502627.921979:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502627.922119:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd509d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562090112:0@lo:38:kworker.0 00000100:00100000:2.0:1777502627.922120:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562089984 00000100:00100000:2.0:1777502627.922134:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562089984 00000100:00100000:0.0:1777502627.922164:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502627.922185:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 414 00000100:00100000:0.0:1777502627.922196:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562090240:0@lo:38:kworker.0 00000100:00100000:2.0:1777502627.922198:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee25c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562089984:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502627.922238:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502627.922261:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee25c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562089984:12345-0@lo:400:kworker.0 Request processed in 67us (281us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502627.922265:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562090112 00000100:00100000:2.0:1777502627.922268:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 414 00000100:00100000:3.0:1777502627.922272:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562090112 00000100:00100000:0.0:1777502627.922294:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562090240 00000100:00100000:2.0:1777502627.922321:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f594380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562089984:0@lo:400:kworker.0 00000100:00100000:0.0:1777502627.922333:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562090240 00000100:00100000:3.0:1777502627.922369:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16700 00000100:00100000:3.0:1777502627.922377:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a80f480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562090112:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502627.922394:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502627.922450:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a80f480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562090112:12345-0@lo:38:kworker.0 Request processed in 72us (287us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502627.922472:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16700 00000100:00080000:2.0:1777502627.922482:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502627.922489:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502627.922494:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd509d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562090112:0@lo:38:kworker.0 00000100:00100000:3.0:1777502627.922499:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16701 00000100:00100000:3.0:1777502627.922508:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a80d880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562090240:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502627.922517:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502627.922561:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a80d880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562090240:12345-0@lo:38:kworker.0 Request processed in 53us (323us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502627.922569:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16701 00000100:00080000:2.0:1777502627.922635:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502627.922639:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502627.922648:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562090240:0@lo:38:kworker.0 00010000:00010000:3.0:1777502628.305729:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502629.329740:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502630.353798:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502631.377948:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502632.401755:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502633.041762:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502633.041769:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502633.041773:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502633.041781:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8755 00000100:00080000:0.0:1777502633.041786:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502633.041793:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502633.041817:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a80ce00 x1863837562090496/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502633.041840:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502633.041844:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502633.041860:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502633.041865:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8755 00000100:00080000:0.0:1777502633.041868:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502633.041873:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502633.041890:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a80ca80 x1863837562090624/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502633.041924:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a80ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562090496:0@lo:38:kworker.0 00000100:00100000:0.0:1777502633.041995:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502633.042153:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a80dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562090368:0@lo:400:kworker.0 00000100:00100000:0.0:1777502633.042239:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562090496 00000100:00100000:3.0:1777502633.042240:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502633.042246:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562090496 00000100:00100000:2.0:1777502633.042276:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562090368 00000100:00100000:2.0:1777502633.042281:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562090368 00000100:00100000:0.0:1777502633.042335:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16702 00000100:00100000:0.0:1777502633.042342:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a80f100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562090496:12345-0@lo:38:kworker.0 00000100:00100000:2.0:1777502633.042350:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 415 00010000:02000000:0.0:1777502633.042372:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502633.042387:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee25450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562090368:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502633.042445:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee25450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562090368:12345-0@lo:400:kworker.0 Request processed in 87us (205us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502633.042454:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 415 00000100:00100000:2.0:1777502633.042526:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a80dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562090368:0@lo:400:kworker.0 00000100:00100000:0.0:1777502633.042527:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a80f100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562090496:12345-0@lo:38:kworker.0 Request processed in 182us (533us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502633.042539:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16702 00000100:00080000:0.0:1777502633.042599:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502633.042606:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502633.042611:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a80ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562090496:0@lo:38:kworker.0 00000100:00100000:0.0:1777502633.042621:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a80ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562090624:0@lo:38:kworker.0 00000100:00100000:0.0:1777502633.042655:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502633.042701:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562090624 00000100:00100000:3.0:1777502633.042706:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562090624 00000100:00100000:0.0:1777502633.042744:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16703 00000100:00100000:0.0:1777502633.042751:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a80fb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562090624:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502633.042761:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502633.042798:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a80fb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562090624:12345-0@lo:38:kworker.0 Request processed in 46us (146us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502633.042809:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16703 00000100:00080000:0.0:1777502633.042843:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502633.042847:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502633.042852:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a80ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562090624:0@lo:38:kworker.0 00010000:00010000:3.0:1777502633.425721:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502634.449791:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502635.473748:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502636.497731:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502637.521751:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502638.161759:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502638.161765:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502638.161768:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502638.161776:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8760 00000100:00080000:3.0:1777502638.161781:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502638.161787:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502638.161807:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a058a80 x1863837562090880/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502638.161823:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502638.161826:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502638.161828:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502638.161832:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8760 00000100:00080000:3.0:1777502638.161835:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502638.161838:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502638.161852:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c6b480 x1863837562091008/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502638.161880:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562090752:0@lo:400:kworker.0 00000100:00100000:3.0:1777502638.161948:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502638.162021:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562090752 00000100:00100000:3.0:1777502638.162027:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562090752 00000100:00100000:3.0:1777502638.162066:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 416 00000100:00100000:3.0:1777502638.162074:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee27450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562090752:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502638.162116:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee27450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562090752:12345-0@lo:400:kworker.0 Request processed in 43us (169us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502638.162143:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562090880:0@lo:38:kworker.0 00000100:00100000:3.0:1777502638.162170:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 416 00000100:00100000:0.0:1777502638.162179:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502638.162202:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562091008:0@lo:38:kworker.0 00000100:00100000:3.0:1777502638.162227:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a05b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562090752:0@lo:400:kworker.0 00000100:00100000:0.0:1777502638.162231:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502638.162270:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562090880 00000100:00100000:3.0:1777502638.162274:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562090880 00000100:00100000:3.0:1777502638.162293:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562091008 00000100:00100000:3.0:1777502638.162296:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562091008 00000100:00100000:3.0:1777502638.162314:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16704 00000100:00100000:3.0:1777502638.162319:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a80ca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562090880:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502638.162332:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502638.162354:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16705 00000100:00100000:0.0:1777502638.162362:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a80ce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562091008:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502638.162363:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a80ca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562090880:12345-0@lo:38:kworker.0 Request processed in 42us (184us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502638.162373:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16704 00010000:02000000:0.0:1777502638.162378:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502638.162424:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a80ce00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562091008:12345-0@lo:38:kworker.0 Request processed in 61us (195us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502638.162432:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16705 00000100:00080000:0.0:1777502638.162470:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502638.162477:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502638.162482:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a058a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562090880:0@lo:38:kworker.0 00000100:00080000:0.0:1777502638.162496:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502638.162500:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502638.162504:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562091008:0@lo:38:kworker.0 00010000:00010000:0.0:1777502638.545757:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502639.569744:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502640.593717:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502641.617740:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502642.641740:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502643.281791:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502643.281810:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502643.281815:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502643.281824:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8765 00000100:00080000:0.0:1777502643.281829:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502643.281836:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502643.281848:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a80d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562091136:0@lo:400:kworker.0 00000100:00100000:0.0:1777502643.281863:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a80ea00 x1863837562091264/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502643.281885:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502643.281891:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502643.281894:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502643.281900:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8765 00000100:00080000:0.0:1777502643.281903:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502643.281908:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502643.281926:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4a80c700 x1863837562091392/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502643.281973:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a80ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562091264:0@lo:38:kworker.0 00000100:00100000:2.0:1777502643.282071:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502643.282117:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502643.282161:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a80c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562091392:0@lo:38:kworker.0 00000100:00100000:2.0:1777502643.282164:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562091136 00000100:00100000:3.0:1777502643.282165:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562091264 00000100:00100000:2.0:1777502643.282173:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562091136 00000100:00100000:3.0:1777502643.282173:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562091264 00000100:00100000:3.0:1777502643.282217:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16706 00000100:00100000:0.0:1777502643.282223:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502643.282228:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a80dc00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562091264:12345-0@lo:38:kworker.0 00000100:00100000:2.0:1777502643.282233:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 417 00000100:00100000:2.0:1777502643.282246:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee26850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562091136:12345-0@lo:400:kworker.0 00010000:02000000:3.0:1777502643.282302:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502643.282306:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee26850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562091136:12345-0@lo:400:kworker.0 Request processed in 62us (235us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502643.282315:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 417 00000100:00100000:3.0:1777502643.282363:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562091392 00000100:00100000:3.0:1777502643.282368:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562091392 00000100:00100000:2.0:1777502643.282403:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a80d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562091136:0@lo:400:kworker.0 00000100:00100000:3.0:1777502643.282413:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16707 00000100:00100000:3.0:1777502643.282423:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a80df80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562091392:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502643.282433:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502643.282483:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a80df80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562091392:12345-0@lo:38:kworker.0 Request processed in 59us (264us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502643.282489:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a80dc00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562091264:12345-0@lo:38:kworker.0 Request processed in 260us (372us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502643.282500:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16706 00000100:00100000:3.0:1777502643.282507:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16707 00000100:00080000:3.0:1777502643.282598:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502643.282602:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502643.282606:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a80ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562091264:0@lo:38:kworker.0 00000100:00080000:3.0:1777502643.282619:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502643.282624:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502643.282627:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a80c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562091392:0@lo:38:kworker.0 00010000:00010000:3.0:1777502643.665770:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502644.689803:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502645.713761:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502646.737852:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502647.761694:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502648.401925:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502648.401931:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502648.401937:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502648.401948:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8770 00000100:00080000:3.0:1777502648.401953:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502648.401963:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502648.401988:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c6b800 x1863837562091648/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502648.402014:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502648.402018:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502648.402021:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502648.402027:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8770 00000100:00080000:3.0:1777502648.402031:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502648.402039:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502648.402071:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c6b480 x1863837562091776/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502648.402113:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562091520:0@lo:400:kworker.0 00000100:00100000:3.0:1777502648.402225:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502648.402326:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562091648:0@lo:38:kworker.0 00000100:00100000:3.0:1777502648.402376:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502648.402411:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562091776:0@lo:38:kworker.0 00000100:00100000:3.0:1777502648.402455:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502648.402567:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562091648 00000100:00100000:0.0:1777502648.402573:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562091648 00000100:00100000:0.0:1777502648.402623:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562091776 00000100:00100000:0.0:1777502648.402628:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562091776 00000100:00100000:0.0:1777502648.402656:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16708 00000100:00100000:0.0:1777502648.402665:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a058a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562091648:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502648.402695:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502648.402743:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a058a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562091648:12345-0@lo:38:kworker.0 Request processed in 77us (374us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502648.402751:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16708 00000100:00100000:0.0:1777502648.402771:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16709 00000100:00100000:0.0:1777502648.402776:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a05b100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562091776:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502648.402784:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502648.402815:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a05b100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562091776:12345-0@lo:38:kworker.0 Request processed in 38us (365us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502648.402823:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16709 00000100:00080000:0.0:1777502648.403518:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502648.403539:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502648.403544:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562091776:0@lo:38:kworker.0 00000100:00080000:0.0:1777502648.403639:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502648.403645:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502648.403649:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562091648:0@lo:38:kworker.0 00000100:00100000:2.0:1777502648.403987:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562091520 00000100:00100000:2.0:1777502648.403992:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562091520 00000100:00100000:2.0:1777502648.404027:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 418 00000100:00100000:2.0:1777502648.404034:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee27850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562091520:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502648.404106:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee27850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562091520:12345-0@lo:400:kworker.0 Request processed in 69us (1880us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502648.404115:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 418 00000100:00100000:3.0:1777502648.404133:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562091520:0@lo:400:kworker.0 00010000:00010000:3.0:1777502648.785732:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502649.809821:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502650.833748:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502651.857713:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502652.881720:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502653.521736:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502653.521742:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1777502653.521744:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562091904:0@lo:400:kworker.0 00000100:00080000:2.0:1777502653.521746:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502653.521753:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8775 00000100:00080000:2.0:1777502653.521757:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502653.521764:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502653.521783:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc44799500 x1863837562092032/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502653.521797:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502653.521800:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502653.521803:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502653.521816:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8775 00000100:00080000:2.0:1777502653.521819:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502653.521824:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502653.521840:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc44798000 x1863837562092160/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502653.521861:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502653.521897:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562091904 00000100:00100000:2.0:1777502653.521903:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562091904 00000100:00100000:2.0:1777502653.521954:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 419 00000100:00100000:2.0:1777502653.521967:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee24050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562091904:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502653.521993:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562092032:0@lo:38:kworker.0 00000100:00100000:2.0:1777502653.522039:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee24050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562091904:12345-0@lo:400:kworker.0 Request processed in 74us (176us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502653.522040:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502653.522062:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 419 00000100:00100000:3.0:1777502653.522075:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44798000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562092160:0@lo:38:kworker.0 00000100:00100000:3.0:1777502653.522107:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502653.522138:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562092032 00000100:00100000:2.0:1777502653.522143:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562092032 00000100:00100000:3.0:1777502653.522152:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562092160 00000100:00100000:3.0:1777502653.522157:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562092160 00000100:00100000:2.0:1777502653.522175:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16710 00000100:00100000:2.0:1777502653.522183:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f713800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562092032:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502653.522195:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16711 00010000:02000000:2.0:1777502653.522198:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502653.522204:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f710380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562092160:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502653.522215:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502653.522250:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f713800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562092032:12345-0@lo:38:kworker.0 Request processed in 65us (210us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502653.522258:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16710 00000100:00100000:3.0:1777502653.522263:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f710380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562092160:12345-0@lo:38:kworker.0 Request processed in 59us (155us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502653.522278:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16711 00000100:00100000:2.0:1777502653.522304:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562091904:0@lo:400:kworker.0 00000100:00080000:3.0:1777502653.522336:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502653.522341:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502653.522345:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562092032:0@lo:38:kworker.0 00000100:00080000:3.0:1777502653.522358:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502653.522363:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502653.522366:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44798000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562092160:0@lo:38:kworker.0 00010000:00010000:3.0:1777502653.905742:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502654.929712:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:02000400:2.0:1777502655.569667:0:221612:0:(genops.c:1873:obd_exports_barrier()) lustre-MDT0000 is waiting for obd_unlinked_exports more than 64 seconds. The obd refcount = 5. Is it stuck? 00000020:02000400:2.0:1777502655.572344:0:221612:0:(genops.c:1840:print_export_data()) lustre-MDT0000: UNLINKED ffff8dfd66b1b800 0775d2ab-558e-41f1-82c0-14244d133390 192.168.202.27@tcp 1 (0 0 0) 1 0 0 0: 0000000000000000 8589949329 stale:0 00010000:00010000:3.0:1777502655.953722:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502656.977747:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502658.001731:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502658.641730:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502658.641739:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502658.641744:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502658.641752:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8780 00000100:00080000:2.0:1777502658.641757:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502658.641765:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502658.641792:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4479b800 x1863837562092416/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502658.641812:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502658.641815:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502658.641818:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502658.641823:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8780 00000100:00080000:2.0:1777502658.641837:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502658.641842:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502658.641859:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc44799f80 x1863837562092544/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502658.641890:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562092288:0@lo:400:kworker.0 00000100:00100000:2.0:1777502658.641970:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502658.642053:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4479b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562092416:0@lo:38:kworker.0 00000100:00100000:2.0:1777502658.642091:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502658.642114:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562092544:0@lo:38:kworker.0 00000100:00100000:3.0:1777502658.642130:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562092288 00000100:00100000:0.0:1777502658.642131:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562092416 00000100:00100000:2.0:1777502658.642139:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502658.642140:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562092416 00000100:00100000:3.0:1777502658.642140:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562092288 00000100:00100000:0.0:1777502658.642176:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562092544 00000100:00100000:0.0:1777502658.642181:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562092544 00000100:00100000:3.0:1777502658.642189:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 420 00000100:00100000:3.0:1777502658.642200:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee26c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562092288:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502658.642213:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16712 00000100:00100000:0.0:1777502658.642221:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc44798a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562092416:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502658.642265:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee26c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562092288:12345-0@lo:400:kworker.0 Request processed in 66us (295us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502658.642273:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 420 00010000:02000000:0.0:1777502658.642282:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502658.642327:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc44798a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562092416:12345-0@lo:38:kworker.0 Request processed in 106us (237us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502658.642342:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16713 00000100:00100000:0.0:1777502658.642350:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16712 00000100:00100000:3.0:1777502658.642352:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4479aa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562092544:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502658.642364:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502658.642402:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4479aa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562092544:12345-0@lo:38:kworker.0 Request processed in 51us (263us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502658.642410:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16713 00000100:00080000:0.0:1777502658.642434:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502658.642440:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502658.642446:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4479b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562092416:0@lo:38:kworker.0 00000100:00080000:0.0:1777502658.642465:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502658.642471:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502658.642476:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562092544:0@lo:38:kworker.0 00000100:00100000:3.0:1777502658.642481:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562092288:0@lo:400:kworker.0 00010000:00010000:0.0:1777502659.025745:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502660.049753:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502661.073726:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502662.097720:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502663.121752:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502663.761800:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502663.761808:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502663.761813:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502663.761826:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8785 00000100:00080000:2.0:1777502663.761831:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502663.761841:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502663.761874:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4479a680 x1863837562092800/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502663.761900:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502663.761905:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502663.761907:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502663.761913:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8785 00000100:00080000:2.0:1777502663.761917:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502663.761924:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502663.761963:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc44799880 x1863837562092928/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502663.762039:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562092672:0@lo:400:kworker.0 00000100:00100000:2.0:1777502663.762150:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502663.762168:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4479a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562092800:0@lo:38:kworker.0 00000100:00100000:0.0:1777502663.762240:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502663.762308:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562092672 00000100:00100000:2.0:1777502663.762318:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562092672 00000100:00100000:0.0:1777502663.762394:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562092928:0@lo:38:kworker.0 00000100:00100000:2.0:1777502663.762413:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 421 00000100:00100000:2.0:1777502663.762432:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee27c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562092672:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502663.762435:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562092800 00000100:00100000:0.0:1777502663.762446:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502663.762448:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562092800 00000100:00100000:3.0:1777502663.762486:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562092928 00000100:00100000:3.0:1777502663.762490:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562092928 00000100:00100000:0.0:1777502663.762507:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16714 00000100:00100000:0.0:1777502663.762518:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a80c700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562092800:12345-0@lo:38:kworker.0 00000100:00100000:2.0:1777502663.762528:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee27c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562092672:12345-0@lo:400:kworker.0 Request processed in 95us (371us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502663.762528:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16715 00000100:00100000:3.0:1777502663.762543:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a80ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562092928:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502663.762547:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502663.762552:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 421 00010000:02000000:3.0:1777502663.762557:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502663.762636:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a80ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562092928:12345-0@lo:38:kworker.0 Request processed in 95us (198us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502663.762643:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a80c700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562092800:12345-0@lo:38:kworker.0 Request processed in 122us (402us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502663.762643:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16715 00000100:00100000:0.0:1777502663.762654:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16714 00000100:00100000:2.0:1777502663.762707:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562092672:0@lo:400:kworker.0 00000100:00080000:3.0:1777502663.762717:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502663.762723:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502663.762728:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4479a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562092800:0@lo:38:kworker.0 00000100:00080000:3.0:1777502663.762742:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502663.762746:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502663.762749:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562092928:0@lo:38:kworker.0 00010000:00010000:3.0:1777502664.145784:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502665.169785:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502666.193744:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502667.217760:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502668.241817:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502668.881771:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502668.881777:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502668.881781:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502668.881790:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8791 00000100:00080000:2.0:1777502668.881794:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502668.881802:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502668.881828:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc44799f80 x1863837562093184/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502668.881848:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502668.881852:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502668.881855:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502668.881860:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8791 00000100:00080000:2.0:1777502668.881877:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502668.881882:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502668.881899:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4479b800 x1863837562093312/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502668.881933:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562093056:0@lo:400:kworker.0 00000100:00100000:2.0:1777502668.882017:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502668.882127:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562093056 00000100:00100000:3.0:1777502668.882128:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562093184:0@lo:38:kworker.0 00000100:00100000:2.0:1777502668.882135:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562093056 00000100:00100000:3.0:1777502668.882173:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502668.882188:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 422 00000100:00100000:2.0:1777502668.882198:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee25850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562093056:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502668.882202:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4479b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562093312:0@lo:38:kworker.0 00000100:00100000:3.0:1777502668.882239:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502668.882259:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee25850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562093056:12345-0@lo:400:kworker.0 Request processed in 62us (242us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502668.882266:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 422 00000100:00100000:3.0:1777502668.882297:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562093184 00000100:00100000:3.0:1777502668.882302:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562093184 00000100:00100000:2.0:1777502668.882317:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562093312 00000100:00100000:2.0:1777502668.882322:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562093312 00000100:00100000:3.0:1777502668.882352:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16716 00000100:00100000:3.0:1777502668.882359:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f712d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562093184:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502668.882375:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502668.882400:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16717 00000100:00100000:2.0:1777502668.882408:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc547b1c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562093312:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502668.882419:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502668.882465:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc547b1c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562093312:12345-0@lo:38:kworker.0 Request processed in 55us (225us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502668.882472:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16717 00000100:00100000:3.0:1777502668.882511:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f712d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562093184:12345-0@lo:38:kworker.0 Request processed in 151us (338us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502668.882520:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16716 00000100:00100000:2.0:1777502668.882539:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562093056:0@lo:400:kworker.0 00000100:00080000:2.0:1777502668.882597:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502668.882602:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502668.882607:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562093184:0@lo:38:kworker.0 00000100:00080000:2.0:1777502668.882630:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502668.882635:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502668.882640:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4479b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562093312:0@lo:38:kworker.0 00010000:00010000:3.0:1777502669.265712:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502670.289724:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502671.313816:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502672.337770:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502673.361818:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502674.001820:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502674.001829:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502674.001836:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502674.001849:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8796 00000100:00080000:0.0:1777502674.001876:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502674.001886:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502674.001925:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c68000 x1863837562093568/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502674.001963:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502674.001970:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502674.001974:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502674.001982:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8796 00000100:00080000:0.0:1777502674.001985:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502674.001991:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502674.002011:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c6b800 x1863837562093696/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502674.002055:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4a80d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562093440:0@lo:400:kworker.0 00000100:00100000:0.0:1777502674.002168:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502674.002231:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562093440 00000100:00100000:3.0:1777502674.002242:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562093440 00000100:00100000:3.0:1777502674.002341:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 423 00000100:00100000:3.0:1777502674.002356:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee24c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562093440:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502674.002441:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee24c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562093440:12345-0@lo:400:kworker.0 Request processed in 88us (276us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502674.002458:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562093568:0@lo:38:kworker.0 00000100:00100000:3.0:1777502674.002463:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 423 00000100:00100000:0.0:1777502674.002807:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4a80d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562093440:0@lo:400:kworker.0 00000100:00100000:0.0:1777502674.002878:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502674.002921:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562093696:0@lo:38:kworker.0 00000100:00100000:0.0:1777502674.002956:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502674.002971:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562093568 00000100:00100000:2.0:1777502674.002976:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562093568 00000100:00100000:3.0:1777502674.002987:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562093696 00000100:00100000:3.0:1777502674.002991:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562093696 00000100:00100000:2.0:1777502674.003012:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16718 00000100:00100000:2.0:1777502674.003019:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c6b480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562093568:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502674.003019:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16719 00000100:00100000:3.0:1777502674.003026:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4a80d180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562093696:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502674.003039:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:3.0:1777502674.003039:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502674.003097:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c6b480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562093568:12345-0@lo:38:kworker.0 Request processed in 77us (221us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502674.003097:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4a80d180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562093696:12345-0@lo:38:kworker.0 Request processed in 68us (143us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502674.003106:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16719 00000100:00100000:2.0:1777502674.003113:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16718 00000100:00080000:0.0:1777502674.003190:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502674.003201:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502674.003210:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562093568:0@lo:38:kworker.0 00000100:00080000:0.0:1777502674.003235:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502674.003242:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502674.003255:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562093696:0@lo:38:kworker.0 00010000:00010000:3.0:1777502674.385719:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502675.409738:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502676.433741:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502677.457747:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502678.481767:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502679.121744:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1777502679.121744:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4479b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562093824:0@lo:400:kworker.0 00000100:00080000:2.0:1777502679.121750:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502679.121755:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502679.121762:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8801 00000100:00080000:2.0:1777502679.121767:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502679.121773:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502679.121801:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc44799f80 x1863837562093952/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502679.121820:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502679.121826:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1777502679.121827:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:2.0:1777502679.121831:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502679.121836:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8801 00000100:00080000:2.0:1777502679.121839:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502679.121844:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502679.121871:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc44799180 x1863837562094080/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502679.121913:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562093824 00000100:00100000:3.0:1777502679.121920:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562093824 00000100:00100000:3.0:1777502679.121970:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 424 00000100:00100000:3.0:1777502679.121982:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f69b850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562093824:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502679.122044:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f69b850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562093824:12345-0@lo:400:kworker.0 Request processed in 66us (221us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502679.122053:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 424 00000100:00100000:3.0:1777502679.122125:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4479b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562093824:0@lo:400:kworker.0 00000100:00100000:3.0:1777502679.122211:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562093952:0@lo:38:kworker.0 00000100:00100000:3.0:1777502679.122253:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502679.122274:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562094080:0@lo:38:kworker.0 00000100:00100000:3.0:1777502679.122295:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502679.122332:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562093952 00000100:00100000:3.0:1777502679.122335:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562093952 00000100:00100000:3.0:1777502679.122357:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562094080 00000100:00100000:3.0:1777502679.122360:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562094080 00000100:00100000:3.0:1777502679.122380:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16720 00000100:00100000:3.0:1777502679.122385:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493faa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562093952:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502679.122398:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502679.122412:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16721 00000100:00100000:2.0:1777502679.122419:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493f8000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562094080:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502679.122429:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493faa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562093952:12345-0@lo:38:kworker.0 Request processed in 41us (174us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502679.122435:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16720 00010000:02000000:2.0:1777502679.122449:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00080000:3.0:1777502679.122471:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502679.122484:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502679.122490:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562093952:0@lo:38:kworker.0 00000100:00100000:2.0:1777502679.122494:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493f8000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562094080:12345-0@lo:38:kworker.0 Request processed in 72us (197us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502679.122504:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16721 00000100:00080000:3.0:1777502679.122511:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502679.122516:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502679.122521:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562094080:0@lo:38:kworker.0 00010000:00010000:0.0:1777502679.505740:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502680.529758:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502681.553771:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502682.577784:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502683.601847:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502684.241865:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502684.241870:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502684.241875:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502684.241884:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8806 00000100:00080000:0.0:1777502684.241889:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502684.241899:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502684.241929:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c68000 x1863837562094336/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502684.241956:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502684.241960:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502684.241964:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502684.241969:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8806 00000100:00080000:0.0:1777502684.241973:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502684.241980:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502684.241996:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4cef3b80 x1863837562094464/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502684.242039:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562094208:0@lo:400:kworker.0 00000100:00100000:0.0:1777502684.242146:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502684.242303:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562094336:0@lo:38:kworker.0 00000100:00100000:0.0:1777502684.242348:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502684.242433:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4cef3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562094464:0@lo:38:kworker.0 00000100:00100000:0.0:1777502684.242467:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502684.242516:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562094208 00000100:00100000:0.0:1777502684.242522:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562094208 00000100:00100000:0.0:1777502684.242627:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 425 00000100:00100000:0.0:1777502684.242643:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f698c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562094208:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502684.242745:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f698c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562094208:12345-0@lo:400:kworker.0 Request processed in 103us (598us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502684.242758:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 425 00000100:00100000:0.0:1777502684.242884:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562094336 00000100:00100000:0.0:1777502684.242889:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562094336 00000100:00100000:0.0:1777502684.242931:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562094464 00000100:00100000:0.0:1777502684.242934:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562094464 00000100:00100000:0.0:1777502684.242988:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16722 00000100:00100000:0.0:1777502684.243001:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef0a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562094336:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502684.243019:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502684.243086:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16723 00000100:00100000:2.0:1777502684.243099:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfccfeae680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562094464:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502684.243119:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502684.243229:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfccfeae680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562094464:12345-0@lo:38:kworker.0 Request processed in 129us (757us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502684.243240:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16723 00000100:00100000:2.0:1777502684.243355:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562094208:0@lo:400:kworker.0 00000100:00080000:2.0:1777502684.243483:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502684.243494:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502684.243500:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4cef3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562094464:0@lo:38:kworker.0 00000100:00100000:0.0:1777502684.246623:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef0a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562094336:12345-0@lo:38:kworker.0 Request processed in 3623us (4276us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502684.246643:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16722 00000100:00080000:2.0:1777502684.246669:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502684.246680:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502684.246688:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562094336:0@lo:38:kworker.0 00010000:00010000:3.0:1777502684.625793:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502685.649792:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502686.673738:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502687.697733:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502688.721725:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502689.361728:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502689.361734:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502689.361739:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502689.361747:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8811 00000100:00080000:0.0:1777502689.361753:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502689.361760:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502689.361784:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfccfeafb80 x1863837562094720/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502689.361819:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502689.361824:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502689.361827:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502689.361832:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8811 00000100:00080000:0.0:1777502689.361835:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502689.361840:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502689.361858:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfccfeadc00 x1863837562094848/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502689.361892:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfccfeafb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562094720:0@lo:38:kworker.0 00000100:00100000:0.0:1777502689.361961:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502689.362024:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfccfeadc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562094848:0@lo:38:kworker.0 00000100:00100000:0.0:1777502689.362054:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502689.362103:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562094720 00000100:00100000:0.0:1777502689.362111:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562094720 00000100:00100000:0.0:1777502689.362162:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562094848 00000100:00100000:0.0:1777502689.362166:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562094848 00000100:00100000:0.0:1777502689.362196:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16724 00000100:00100000:0.0:1777502689.362204:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfccfeac000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562094720:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502689.362222:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502689.362273:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfccfeac000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562094720:12345-0@lo:38:kworker.0 Request processed in 67us (313us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502689.362281:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16724 00000100:00100000:0.0:1777502689.362305:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16725 00000100:00100000:0.0:1777502689.362311:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfccfeae300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562094848:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502689.362320:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502689.362361:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfccfeae300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562094848:12345-0@lo:38:kworker.0 Request processed in 49us (307us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502689.362369:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16725 00000100:00100000:0.0:1777502689.362411:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfccfeaca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562094592:0@lo:400:kworker.0 00000100:00100000:0.0:1777502689.362451:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502689.362517:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562094592 00000100:00100000:0.0:1777502689.362521:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562094592 00000100:00100000:0.0:1777502689.362560:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 426 00000100:00100000:0.0:1777502689.362569:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f698050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562094592:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502689.362639:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f698050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562094592:12345-0@lo:400:kworker.0 Request processed in 71us (189us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502689.362648:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 426 00000100:00100000:0.0:1777502689.362710:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfccfeaca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562094592:0@lo:400:kworker.0 00000100:00080000:3.0:1777502689.362768:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502689.362774:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502689.362779:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfccfeafb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562094720:0@lo:38:kworker.0 00000100:00080000:3.0:1777502689.362796:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502689.362801:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502689.362805:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfccfeadc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562094848:0@lo:38:kworker.0 00010000:00010000:3.0:1777502689.745766:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502690.769919:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502691.793774:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502692.817750:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502693.841720:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502694.481861:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502694.481868:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502694.481873:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502694.481881:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8816 00000100:00080000:0.0:1777502694.481886:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502694.481894:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502694.481921:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfccfead500 x1863837562095104/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502694.481943:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502694.481947:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502694.481950:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502694.481970:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8816 00000100:00080000:0.0:1777502694.481973:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502694.481978:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502694.482001:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfccfead180 x1863837562095232/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502694.482035:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfccfead500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562095104:0@lo:38:kworker.0 00000100:00100000:0.0:1777502694.482119:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502694.482173:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfccfeaca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562094976:0@lo:400:kworker.0 00000100:00100000:0.0:1777502694.482178:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfccfead180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562095232:0@lo:38:kworker.0 00000100:00100000:2.0:1777502694.482903:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562095104 00000100:00100000:2.0:1777502694.482912:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562095104 00000100:00100000:2.0:1777502694.482952:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16726 00000100:00100000:2.0:1777502694.482960:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfccfeaf100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562095104:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502694.482986:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502694.483061:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502694.483094:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502694.483137:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562095232 00000100:00100000:0.0:1777502694.483142:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562095232 00000100:00100000:0.0:1777502694.483170:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16727 00000100:00100000:0.0:1777502694.483177:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfccfeadc00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562095232:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502694.483187:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502694.483219:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfccfeadc00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562095232:12345-0@lo:38:kworker.0 Request processed in 42us (159us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502694.483225:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16727 00000100:00080000:0.0:1777502694.483266:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502694.483271:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502694.483274:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfccfead180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562095232:0@lo:38:kworker.0 00000100:00100000:0.0:1777502694.483339:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562094976 00000100:00100000:0.0:1777502694.483342:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562094976 00000100:00100000:0.0:1777502694.483371:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 427 00000100:00100000:0.0:1777502694.483379:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f698450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562094976:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502694.483418:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f698450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562094976:12345-0@lo:400:kworker.0 Request processed in 41us (327us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502694.483428:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 427 00000100:00100000:0.0:1777502694.483503:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfccfeaca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562094976:0@lo:400:kworker.0 00000100:00100000:2.0:1777502694.483672:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfccfeaf100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562095104:12345-0@lo:38:kworker.0 Request processed in 713us (1554us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502694.483678:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16726 00000100:00080000:0.0:1777502694.483802:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502694.483807:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502694.483813:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfccfead500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562095104:0@lo:38:kworker.0 00010000:00010000:3.0:1777502694.865760:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502695.889711:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502696.913746:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502697.937718:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502698.961763:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502699.601786:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502699.601792:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502699.601795:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502699.601801:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8821 00000100:00080000:2.0:1777502699.601805:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502699.601811:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502699.601829:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc44799500 x1863837562095488/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502699.601841:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502699.601844:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502699.601845:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502699.601849:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8821 00000100:00080000:2.0:1777502699.601851:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502699.601854:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502699.601863:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc44799180 x1863837562095616/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502699.601884:0:3310:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1777502699.601899:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44798000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562095360:0@lo:400:kworker.0 00000100:00100000:2.0:1777502699.601965:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502699.602347:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562095488:0@lo:38:kworker.0 00000100:00100000:0.0:1777502699.602387:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502699.602415:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562095616:0@lo:38:kworker.0 00000100:00100000:2.0:1777502699.602435:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562095488 00000100:00100000:0.0:1777502699.602442:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502699.602443:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562095488 00000100:00100000:0.0:1777502699.602487:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562095360 00000100:00100000:0.0:1777502699.602492:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562095360 00000100:00100000:2.0:1777502699.602495:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562095616 00000100:00100000:2.0:1777502699.602499:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562095616 00000100:00100000:2.0:1777502699.602524:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16728 00000100:00100000:2.0:1777502699.602531:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfccfead500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562095488:12345-0@lo:38:kworker.0 00000100:00100000:0.0:1777502699.602540:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 428 00010000:02000000:2.0:1777502699.602545:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502699.602551:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f69ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562095360:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502699.602706:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16729 00000100:00100000:3.0:1777502699.602720:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfccfeaca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562095616:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502699.602731:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502699.603067:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfccfead500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562095488:12345-0@lo:38:kworker.0 Request processed in 535us (681us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502699.603073:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16728 00000100:00100000:3.0:1777502699.603090:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfccfeaca80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562095616:12345-0@lo:38:kworker.0 Request processed in 373us (647us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502699.603094:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f69ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562095360:12345-0@lo:400:kworker.0 Request processed in 546us (1128us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502699.603098:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16729 00000100:00100000:0.0:1777502699.603102:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 428 00000100:00100000:3.0:1777502699.603201:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44798000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562095360:0@lo:400:kworker.0 00000100:00080000:0.0:1777502699.603365:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502699.603371:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502699.603376:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562095488:0@lo:38:kworker.0 00000100:00080000:0.0:1777502699.603391:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502699.603395:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502699.603399:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562095616:0@lo:38:kworker.0 00010000:00010000:0.0:1777502699.985724:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502701.009746:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502702.033755:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502703.057752:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502704.081767:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502704.721861:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502704.721869:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502704.721873:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502704.721883:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8826 00000100:00080000:3.0:1777502704.721888:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502704.721895:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502704.721919:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc493fa680 x1863837562095872/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502704.721934:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502704.721936:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502704.721938:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502704.721942:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8826 00000100:00080000:3.0:1777502704.721944:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502704.721947:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502704.721970:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc493f8a80 x1863837562096000/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502704.722015:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493fb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562095744:0@lo:400:kworker.0 00000100:00100000:0.0:1777502704.722073:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493fa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562095872:0@lo:38:kworker.0 00000100:00100000:3.0:1777502704.722082:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502704.722114:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502704.722146:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493f8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562096000:0@lo:38:kworker.0 00000100:00100000:0.0:1777502704.722168:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502704.722197:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562095872 00000100:00100000:3.0:1777502704.722205:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562095872 00000100:00100000:3.0:1777502704.722239:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562096000 00000100:00100000:3.0:1777502704.722243:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562096000 00000100:00100000:0.0:1777502704.722248:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562095744 00000100:00100000:0.0:1777502704.722253:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562095744 00000100:00100000:3.0:1777502704.722268:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16730 00000100:00100000:3.0:1777502704.722278:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfccfead180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562095872:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502704.722295:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502704.722301:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 429 00000100:00100000:0.0:1777502704.722311:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f699c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562095744:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502704.722342:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16731 00000100:00100000:2.0:1777502704.722349:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfccfeafb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562096000:12345-0@lo:38:kworker.0 00000100:00100000:0.0:1777502704.722354:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f699c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562095744:12345-0@lo:400:kworker.0 Request processed in 45us (275us total) trans 0 rc 0/0 00010000:02000000:2.0:1777502704.722362:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502704.722363:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 429 00000100:00100000:2.0:1777502704.722417:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfccfeafb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562096000:12345-0@lo:38:kworker.0 Request processed in 65us (247us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502704.722425:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16731 00000100:00080000:0.0:1777502704.722474:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1777502704.722479:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfccfead180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562095872:12345-0@lo:38:kworker.0 Request processed in 199us (363us total) trans 0 rc -19/-19 00000100:00080000:0.0:1777502704.722483:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502704.722492:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493f8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562096000:0@lo:38:kworker.0 00000100:00100000:2.0:1777502704.722493:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16730 00000100:00080000:0.0:1777502704.722526:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502704.722530:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502704.722534:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493fa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562095872:0@lo:38:kworker.0 00000100:00100000:0.0:1777502704.722571:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493fb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562095744:0@lo:400:kworker.0 00010000:00010000:3.0:1777502705.105704:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502706.129764:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502707.153721:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502708.177730:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502709.201763:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502709.841767:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502709.841773:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502709.841777:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502709.841786:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8831 00000100:00080000:2.0:1777502709.841791:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502709.841798:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502709.841823:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc4479b800 x1863837562096256/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502709.841843:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502709.841846:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502709.841862:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502709.841867:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8831 00000100:00080000:2.0:1777502709.841871:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502709.841876:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502709.841893:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc44799880 x1863837562096384/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502709.841973:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:2.0:1777502709.841991:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562096128:0@lo:400:kworker.0 00000100:00100000:2.0:1777502709.842095:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502709.842174:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562096128 00000100:00100000:2.0:1777502709.842182:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562096128 00000100:00100000:2.0:1777502709.842232:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 430 00000100:00100000:2.0:1777502709.842243:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f699050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562096128:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502709.842255:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc4479b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562096256:0@lo:38:kworker.0 00000100:00100000:2.0:1777502709.842386:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f699050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562096128:12345-0@lo:400:kworker.0 Request processed in 144us (291us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502709.842396:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 430 00000100:00100000:2.0:1777502709.842463:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562096128:0@lo:400:kworker.0 00000100:00100000:3.0:1777502709.842492:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502709.842520:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562096384:0@lo:38:kworker.0 00000100:00100000:0.0:1777502709.842546:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562096256 00000100:00100000:3.0:1777502709.842553:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502709.842596:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562096256 00000100:00100000:0.0:1777502709.842620:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562096384 00000100:00100000:0.0:1777502709.842622:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562096384 00000100:00100000:0.0:1777502709.842649:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16732 00000100:00100000:0.0:1777502709.842655:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493f9880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562096256:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502709.842657:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16733 00000100:00100000:3.0:1777502709.842666:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493f8e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562096384:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502709.842672:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:3.0:1777502709.842679:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502709.842710:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493f9880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562096256:12345-0@lo:38:kworker.0 Request processed in 54us (219us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502709.842717:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16732 00000100:00100000:3.0:1777502709.842729:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493f8e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562096384:12345-0@lo:38:kworker.0 Request processed in 60us (177us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502709.842737:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16733 00000100:00080000:3.0:1777502709.842779:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502709.842785:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502709.842793:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562096384:0@lo:38:kworker.0 00000100:00080000:3.0:1777502709.842818:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502709.842821:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502709.842824:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc4479b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562096256:0@lo:38:kworker.0 00010000:00010000:3.0:1777502710.225731:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502711.249780:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502712.273829:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502713.297708:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502714.321740:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502714.961738:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502714.961745:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502714.961749:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502714.961754:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8837 00000100:00080000:3.0:1777502714.961762:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502714.961769:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502714.961790:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc493fb800 x1863837562096640/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502714.961804:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502714.961806:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502714.961808:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502714.961812:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8837 00000100:00080000:3.0:1777502714.961813:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502714.961816:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502714.961834:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc493fa680 x1863837562096768/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502714.961862:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1777502714.961933:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562096512:0@lo:400:kworker.0 00000100:00100000:3.0:1777502714.961992:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502714.962045:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493fb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562096640:0@lo:38:kworker.0 00000100:00100000:3.0:1777502714.962069:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502714.962104:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493fa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562096768:0@lo:38:kworker.0 00000100:00100000:2.0:1777502714.962105:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562096640 00000100:00100000:2.0:1777502714.962113:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562096640 00000100:00100000:3.0:1777502714.962147:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502714.962149:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562096512 00000100:00100000:0.0:1777502714.962155:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562096512 00000100:00100000:2.0:1777502714.962158:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562096768 00000100:00100000:2.0:1777502714.962163:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562096768 00000100:00100000:2.0:1777502714.962196:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16734 00000100:00100000:0.0:1777502714.962208:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 431 00000100:00100000:3.0:1777502714.962208:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16735 00000100:00100000:2.0:1777502714.962209:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493f8a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562096640:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502714.962218:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4fa34000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562096768:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502714.962228:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:2.0:1777502714.962256:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502714.962274:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4fa34000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562096768:12345-0@lo:38:kworker.0 Request processed in 54us (131us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502714.962282:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16735 00000100:00100000:2.0:1777502714.962303:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493f8a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562096640:12345-0@lo:38:kworker.0 Request processed in 96us (230us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502714.962312:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16734 00000100:00080000:3.0:1777502714.962331:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502714.962343:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502714.962347:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493fb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562096640:0@lo:38:kworker.0 00000100:00080000:3.0:1777502714.962360:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502714.962363:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502714.962366:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493fa680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562096768:0@lo:38:kworker.0 00000100:00100000:0.0:1777502714.962480:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f69a850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562096512:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502714.962566:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f69a850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562096512:12345-0@lo:400:kworker.0 Request processed in 291us (573us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502714.962597:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 431 00000100:00100000:2.0:1777502714.962619:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562096512:0@lo:400:kworker.0 00010000:00010000:3.0:1777502715.345731:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502716.369750:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502717.393720:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502718.417735:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502719.441713:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502720.081755:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502720.081762:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502720.081767:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502720.081776:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8842 00000100:00080000:0.0:1777502720.081781:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502720.081789:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502720.081813:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd6e6e9500 x1863837562097024/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502720.081845:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502720.081850:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502720.081852:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502720.081870:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8842 00000100:00080000:0.0:1777502720.081873:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502720.081879:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502720.081896:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd6e6eb480 x1863837562097152/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502720.081931:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6e9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562096896:0@lo:400:kworker.0 00000100:00100000:0.0:1777502720.082005:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502720.082080:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562096896 00000100:00100000:2.0:1777502720.082089:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562096896 00000100:00100000:2.0:1777502720.082136:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 432 00000100:00100000:3.0:1777502720.082144:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6e9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562097024:0@lo:38:kworker.0 00000100:00100000:2.0:1777502720.082148:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f69b050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562096896:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502720.082203:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502720.082210:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f69b050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562096896:12345-0@lo:400:kworker.0 Request processed in 65us (205us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502720.082218:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 432 00000100:00100000:3.0:1777502720.082260:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6eb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562097152:0@lo:38:kworker.0 00000100:00100000:0.0:1777502720.082374:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6e9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562096896:0@lo:400:kworker.0 00000100:00100000:3.0:1777502720.082404:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502720.082447:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562097024 00000100:00100000:3.0:1777502720.082450:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562097024 00000100:00100000:3.0:1777502720.082470:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562097152 00000100:00100000:3.0:1777502720.082472:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562097152 00000100:00100000:3.0:1777502720.082538:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16736 00000100:00100000:2.0:1777502720.082547:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16737 00000100:00100000:3.0:1777502720.082549:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4fa36a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562097024:12345-0@lo:38:kworker.0 00000100:00100000:2.0:1777502720.082564:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4fa34e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562097152:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502720.082566:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:2.0:1777502720.082575:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502720.082627:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4fa34e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562097152:12345-0@lo:38:kworker.0 Request processed in 73us (224us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502720.082632:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16737 00000100:00100000:3.0:1777502720.082637:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4fa36a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562097024:12345-0@lo:38:kworker.0 Request processed in 89us (433us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502720.082647:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16736 00000100:00080000:2.0:1777502720.082669:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502720.082674:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502720.082678:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6e9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562097024:0@lo:38:kworker.0 00000100:00080000:2.0:1777502720.082693:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502720.082697:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502720.082701:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6eb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562097152:0@lo:38:kworker.0 00010000:00010000:0.0:1777502720.465733:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502721.489727:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502722.513761:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502723.537719:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502724.561723:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502725.201872:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502725.201878:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502725.201882:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502725.201889:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8847 00000100:00080000:3.0:1777502725.201892:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502725.201898:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502725.201917:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa37480 x1863837562097408/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502725.201930:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502725.201932:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502725.201934:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502725.201937:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8847 00000100:00080000:3.0:1777502725.201939:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502725.201992:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502725.202012:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa35c00 x1863837562097536/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502725.202039:0:3309:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1777502725.202057:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa37100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562097280:0@lo:400:kworker.0 00000100:00100000:3.0:1777502725.202146:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502725.202195:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa37480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562097408:0@lo:38:kworker.0 00000100:00100000:3.0:1777502725.202221:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562097280 00000100:00100000:3.0:1777502725.202229:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562097280 00000100:00100000:2.0:1777502725.202238:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502725.202263:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562097536:0@lo:38:kworker.0 00000100:00100000:3.0:1777502725.202277:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 433 00000100:00100000:3.0:1777502725.202287:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f698850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562097280:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502725.202294:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502725.202344:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f698850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562097280:12345-0@lo:400:kworker.0 Request processed in 59us (199us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502725.202349:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 433 00000100:00100000:2.0:1777502725.202395:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562097408 00000100:00100000:2.0:1777502725.202425:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562097408 00000100:00100000:3.0:1777502725.202436:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa37100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562097280:0@lo:400:kworker.0 00000100:00100000:2.0:1777502725.202452:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562097536 00000100:00100000:2.0:1777502725.202457:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562097536 00000100:00100000:2.0:1777502725.202483:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16738 00000100:00100000:2.0:1777502725.202489:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc44799f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562097408:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502725.202503:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502725.202509:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16739 00000100:00100000:3.0:1777502725.202520:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4479a680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562097536:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502725.202531:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502725.202539:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc44799f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562097408:12345-0@lo:38:kworker.0 Request processed in 49us (302us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502725.202545:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16738 00000100:00100000:3.0:1777502725.202571:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4479a680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562097536:12345-0@lo:38:kworker.0 Request processed in 51us (279us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502725.202598:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16739 00000100:00080000:2.0:1777502725.202600:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502725.202607:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502725.202612:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa37480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562097408:0@lo:38:kworker.0 00000100:00080000:2.0:1777502725.202629:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502725.202635:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502725.202640:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562097536:0@lo:38:kworker.0 00010000:00010000:3.0:1777502725.585726:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502726.609713:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502727.633733:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502728.657808:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502729.681800:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502730.321796:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502730.321805:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502730.321811:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502730.321823:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8852 00000100:00080000:0.0:1777502730.321829:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1777502730.321837:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6e9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562097664:0@lo:400:kworker.0 00000100:00080000:0.0:1777502730.321842:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502730.321874:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd6e6ebb80 x1863837562097792/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502730.321899:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502730.321904:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502730.321908:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502730.321916:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8852 00000100:00080000:0.0:1777502730.321921:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502730.321930:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502730.321937:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502730.322067:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6ebb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562097792:0@lo:38:kworker.0 00000100:00100000:0.0:1777502730.322136:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd6e6e8380 x1863837562097920/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502730.322137:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502730.322193:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6e8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562097920:0@lo:38:kworker.0 00000100:00100000:0.0:1777502730.322254:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562097792 00000100:00100000:0.0:1777502730.322261:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562097792 00000100:00100000:0.0:1777502730.322314:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16740 00000100:00100000:0.0:1777502730.322325:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4479b800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562097792:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502730.322341:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502730.322419:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4479b800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562097792:12345-0@lo:38:kworker.0 Request processed in 96us (293us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502730.322429:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16740 00000100:00100000:0.0:1777502730.322478:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562097664 00000100:00100000:0.0:1777502730.322485:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562097664 00000100:00100000:0.0:1777502730.322544:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 434 00000100:00100000:0.0:1777502730.322556:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f699450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562097664:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502730.322692:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f699450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562097664:12345-0@lo:400:kworker.0 Request processed in 138us (758us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502730.322704:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 434 00000100:00100000:2.0:1777502730.322852:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502730.322859:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6e9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562097664:0@lo:400:kworker.0 00000100:00080000:2.0:1777502730.322918:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502730.322926:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502730.322933:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6ebb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562097792:0@lo:38:kworker.0 00000100:00100000:2.0:1777502730.322995:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562097920 00000100:00100000:2.0:1777502730.323010:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562097920 00000100:00100000:2.0:1777502730.323057:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16741 00000100:00100000:2.0:1777502730.323067:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc44799880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562097920:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502730.323083:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502730.323142:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc44799880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562097920:12345-0@lo:38:kworker.0 Request processed in 75us (296us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502730.323150:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16741 00000100:00080000:2.0:1777502730.323211:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502730.323220:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502730.323229:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6e8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562097920:0@lo:38:kworker.0 00010000:00010000:3.0:1777502730.705796:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502731.729828:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502732.753783:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502733.777781:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502734.801763:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502735.441904:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502735.441923:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502735.441931:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502735.441947:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8857 00000100:00080000:2.0:1777502735.441953:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502735.441966:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502735.442037:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc44799500 x1863837562098176/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502735.442062:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502735.442066:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502735.442069:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502735.442087:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8857 00000100:00080000:2.0:1777502735.442090:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502735.442097:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502735.442118:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc44798000 x1863837562098304/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502735.442196:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562098048:0@lo:400:kworker.0 00000100:00100000:2.0:1777502735.442330:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502735.442479:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562098048 00000100:00100000:3.0:1777502735.442489:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562098048 00000100:00100000:0.0:1777502735.442884:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44799500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562098176:0@lo:38:kworker.0 00000100:00100000:0.0:1777502735.442985:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502735.443037:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc44798000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562098304:0@lo:38:kworker.0 00000100:00100000:0.0:1777502735.443099:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502735.443741:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562098176 00000100:00100000:2.0:1777502735.443752:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562098176 00000100:00100000:2.0:1777502735.443802:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562098304 00000100:00100000:2.0:1777502735.443810:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562098304 00000100:00100000:3.0:1777502735.443820:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 435 00000100:00100000:3.0:1777502735.443843:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f69a050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562098048:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502735.443859:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16742 00000100:00100000:2.0:1777502735.443871:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6e9180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562098176:12345-0@lo:38:kworker.0 00000100:00100000:0.0:1777502735.443874:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16743 00000100:00100000:0.0:1777502735.443889:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6ea300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562098304:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502735.443896:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:0.0:1777502735.443912:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502735.443937:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f69a050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562098048:12345-0@lo:400:kworker.0 Request processed in 101us (1605us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502735.443949:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 435 00000100:00100000:2.0:1777502735.443985:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6e9180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562098176:12345-0@lo:38:kworker.0 Request processed in 112us (997us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502735.443998:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16742 00000100:00100000:0.0:1777502735.444014:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6ea300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562098304:12345-0@lo:38:kworker.0 Request processed in 121us (910us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502735.444024:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16743 00000100:00080000:2.0:1777502735.444083:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502735.444094:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502735.444102:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562098176:0@lo:38:kworker.0 00000100:00100000:3.0:1777502735.444106:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44799180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562098048:0@lo:400:kworker.0 00000100:00080000:2.0:1777502735.444138:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502735.444147:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502735.444155:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc44798000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562098304:0@lo:38:kworker.0 00010000:00010000:3.0:1777502735.825887:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502736.849793:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502737.873750:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502738.897775:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502739.921761:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502740.561827:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502740.561835:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502740.561841:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502740.561856:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8862 00000100:00080000:3.0:1777502740.561864:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502740.561878:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502740.561912:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa36d80 x1863837562098560/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502740.561939:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502740.561945:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502740.561948:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502740.561955:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8862 00000100:00080000:3.0:1777502740.561959:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502740.561983:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502740.562003:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa35500 x1863837562098688/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502740.562045:0:3310:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1777502740.562071:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa37100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562098432:0@lo:400:kworker.0 00000100:00100000:3.0:1777502740.562286:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502740.562333:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa36d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562098560:0@lo:38:kworker.0 00000100:00100000:3.0:1777502740.562406:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562098432 00000100:00100000:3.0:1777502740.562414:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562098432 00000100:00100000:2.0:1777502740.562462:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502740.562470:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 436 00000100:00100000:3.0:1777502740.562482:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f69b450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562098432:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502740.562526:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562098688:0@lo:38:kworker.0 00000100:00100000:3.0:1777502740.562568:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f69b450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562098432:12345-0@lo:400:kworker.0 Request processed in 86us (284us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502740.562619:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502740.562629:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 436 00000100:00100000:2.0:1777502740.562723:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562098560 00000100:00100000:2.0:1777502740.562731:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562098560 00000100:00100000:3.0:1777502740.562734:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa37100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562098432:0@lo:400:kworker.0 00000100:00100000:2.0:1777502740.562788:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562098688 00000100:00100000:2.0:1777502740.562798:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562098688 00000100:00100000:2.0:1777502740.562850:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16744 00000100:00100000:2.0:1777502740.562860:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc44798000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562098560:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502740.562872:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16745 00000100:00100000:3.0:1777502740.562881:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc44799500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562098688:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502740.564006:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:3.0:1777502740.564008:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502740.564100:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc44799500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562098688:12345-0@lo:38:kworker.0 Request processed in 1215us (1517us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502740.564108:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc44798000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562098560:12345-0@lo:38:kworker.0 Request processed in 1242us (1643us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502740.564118:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16745 00000100:00100000:2.0:1777502740.564132:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16744 00000100:00080000:0.0:1777502740.564158:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502740.564166:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502740.564177:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa36d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562098560:0@lo:38:kworker.0 00000100:00080000:0.0:1777502740.564199:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502740.564206:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502740.564214:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562098688:0@lo:38:kworker.0 00010000:00010000:0.0:1777502740.945756:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502741.969800:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502742.993845:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502744.017798:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502745.041743:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502745.681741:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502745.681747:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502745.681752:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1777502745.681754:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa37100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562098816:0@lo:400:kworker.0 00000100:00080000:3.0:1777502745.681785:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8867 00000100:00080000:3.0:1777502745.681788:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502745.681794:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502745.681813:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa35500 x1863837562098944/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502745.681829:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:3.0:1777502745.681829:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502745.681835:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502745.681836:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502745.681840:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8867 00000100:00080000:3.0:1777502745.681842:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502745.681845:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502745.681855:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa36d80 x1863837562099072/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502745.681892:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562098816 00000100:00100000:3.0:1777502745.681898:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562098816 00000100:00100000:2.0:1777502745.681930:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562098944:0@lo:38:kworker.0 00000100:00100000:3.0:1777502745.681940:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 437 00000100:00100000:3.0:1777502745.681949:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f699850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562098816:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502745.681969:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502745.682005:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa36d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562099072:0@lo:38:kworker.0 00000100:00100000:3.0:1777502745.682025:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f699850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562098816:12345-0@lo:400:kworker.0 Request processed in 77us (198us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502745.682033:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 437 00000100:00100000:2.0:1777502745.682041:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502745.682094:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa37100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562098816:0@lo:400:kworker.0 00000100:00100000:0.0:1777502745.682204:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562098944 00000100:00100000:0.0:1777502745.682210:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562098944 00000100:00100000:2.0:1777502745.682257:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562099072 00000100:00100000:0.0:1777502745.682266:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16746 00000100:00100000:2.0:1777502745.682270:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562099072 00000100:00100000:0.0:1777502745.682277:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc44799180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562098944:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502745.682297:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502745.682335:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16747 00000100:00100000:2.0:1777502745.682347:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4cef3b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562099072:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502745.682364:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502745.682441:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4cef3b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562099072:12345-0@lo:38:kworker.0 Request processed in 86us (391us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502745.682452:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16747 00000100:00080000:3.0:1777502745.682505:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502745.682513:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502745.682522:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa36d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562099072:0@lo:38:kworker.0 00000100:00100000:0.0:1777502745.682704:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc44799180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562098944:12345-0@lo:38:kworker.0 Request processed in 424us (730us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502745.682717:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16746 00000100:00080000:3.0:1777502745.682760:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502745.682773:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502745.682784:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562098944:0@lo:38:kworker.0 00010000:00010000:3.0:1777502746.065819:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502747.089737:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502748.113718:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502749.137792:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502750.161804:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502750.802044:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502750.802051:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502750.802058:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502750.802072:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8872 00000100:00080000:0.0:1777502750.802078:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502750.802088:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502750.802249:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd6e6e8e00 x1863837562099328/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502750.802286:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502750.802292:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502750.802305:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502750.802315:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8872 00000100:00080000:0.0:1777502750.802319:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502750.802328:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502750.802347:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd6e6eb800 x1863837562099456/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502750.802558:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:3.0:1777502750.802615:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6ead80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562099200:0@lo:400:kworker.0 00000100:00100000:3.0:1777502750.802706:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502750.802749:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6e8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562099328:0@lo:38:kworker.0 00000100:00100000:3.0:1777502750.802785:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562099200 00000100:00100000:3.0:1777502750.802794:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562099200 00000100:00100000:0.0:1777502750.802833:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502750.802859:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 438 00000100:00100000:0.0:1777502750.802867:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6eb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562099456:0@lo:38:kworker.0 00000100:00100000:3.0:1777502750.802876:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45ccd050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562099200:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502750.802907:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502750.802923:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562099328 00000100:00100000:2.0:1777502750.802930:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562099328 00000100:00100000:3.0:1777502750.802946:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45ccd050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562099200:12345-0@lo:400:kworker.0 Request processed in 75us (238us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502750.802957:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 438 00000100:00100000:2.0:1777502750.802986:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562099456 00000100:00100000:2.0:1777502750.802991:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562099456 00000100:00100000:2.0:1777502750.803035:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16748 00000100:00100000:2.0:1777502750.803044:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6e9c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562099328:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502750.803051:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6ead80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562099200:0@lo:400:kworker.0 00010000:02000000:2.0:1777502750.803066:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502750.803170:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6e9c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562099328:12345-0@lo:38:kworker.0 Request processed in 125us (338us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502750.803179:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16748 00000100:00100000:2.0:1777502750.803207:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16749 00000100:00100000:2.0:1777502750.803214:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6e8380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562099456:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502750.803222:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502750.803266:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6e8380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562099456:12345-0@lo:38:kworker.0 Request processed in 50us (363us total) trans 0 rc -19/-19 00000100:00080000:0.0:1777502750.803270:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1777502750.803275:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16749 00000100:00080000:0.0:1777502750.803279:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502750.803290:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6e8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562099328:0@lo:38:kworker.0 00000100:00080000:0.0:1777502750.803321:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502750.803326:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502750.803334:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6eb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562099456:0@lo:38:kworker.0 00010000:00010000:3.0:1777502751.185824:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502752.209781:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502753.233729:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502754.257781:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502755.282818:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502755.921725:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502755.921731:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502755.921735:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502755.921743:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8878 00000100:00080000:2.0:1777502755.921748:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502755.921755:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502755.921756:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:0.0:1777502755.921786:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562099584:0@lo:400:kworker.0 00000100:00100000:2.0:1777502755.921793:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c6b800 x1863837562099712/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502755.921822:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502755.921827:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502755.921829:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502755.921834:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8878 00000100:00080000:2.0:1777502755.921838:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502755.921843:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502755.921861:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd509d3800 x1863837562099840/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502755.921915:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502755.921922:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562099712:0@lo:38:kworker.0 00000100:00100000:2.0:1777502755.922004:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502755.922037:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd509d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562099840:0@lo:38:kworker.0 00000100:00100000:2.0:1777502755.922073:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502755.922118:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562099712 00000100:00100000:2.0:1777502755.922125:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562099712 00000100:00100000:3.0:1777502755.922146:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562099584 00000100:00100000:3.0:1777502755.922153:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562099584 00000100:00100000:2.0:1777502755.922161:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562099840 00000100:00100000:2.0:1777502755.922165:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562099840 00000100:00100000:2.0:1777502755.922199:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16750 00000100:00100000:3.0:1777502755.922208:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 439 00000100:00100000:2.0:1777502755.922210:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f594380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562099712:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502755.922223:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45cce450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562099584:12345-0@lo:400:kworker.0 00010000:02000000:2.0:1777502755.922230:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502755.922261:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16751 00000100:00100000:0.0:1777502755.922273:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc560c9f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562099840:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502755.922282:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45cce450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562099584:12345-0@lo:400:kworker.0 Request processed in 63us (367us total) trans 0 rc 0/0 00010000:02000000:0.0:1777502755.922286:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502755.922301:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 439 00000100:00100000:0.0:1777502755.922335:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc560c9f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562099840:12345-0@lo:38:kworker.0 Request processed in 65us (261us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502755.922351:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16751 00000100:00100000:0.0:1777502755.922396:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c68000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562099584:0@lo:400:kworker.0 00000100:00100000:2.0:1777502755.922444:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f594380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562099712:12345-0@lo:38:kworker.0 Request processed in 229us (438us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502755.922452:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16750 00000100:00080000:2.0:1777502755.922557:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502755.922562:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502755.922598:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562099712:0@lo:38:kworker.0 00000100:00080000:2.0:1777502755.922610:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502755.922613:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502755.922616:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd509d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562099840:0@lo:38:kworker.0 00010000:00010000:3.0:1777502756.305743:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502757.329819:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502758.353741:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502759.377774:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502760.401726:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502761.041777:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502761.041792:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502761.041796:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502761.041804:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8883 00000100:00080000:0.0:1777502761.041809:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502761.041821:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502761.041849:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd6e6e8e00 x1863837562100096/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502761.041866:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502761.041869:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502761.041871:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502761.041875:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8883 00000100:00080000:0.0:1777502761.041877:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502761.041881:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502761.041894:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd6e6ebb80 x1863837562100224/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502761.041943:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6eb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562099968:0@lo:400:kworker.0 00000100:00100000:0.0:1777502761.042023:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502761.042144:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562099968 00000100:00100000:3.0:1777502761.042152:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562099968 00000100:00100000:0.0:1777502761.042180:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6e8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562100096:0@lo:38:kworker.0 00000100:00100000:0.0:1777502761.042216:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502761.042217:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 440 00000100:00100000:3.0:1777502761.042230:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45ccf850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562099968:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502761.042244:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6ebb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562100224:0@lo:38:kworker.0 00000100:00100000:0.0:1777502761.042285:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502761.042296:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45ccf850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562099968:12345-0@lo:400:kworker.0 Request processed in 68us (271us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502761.042305:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 440 00000100:00100000:3.0:1777502761.042369:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562100096 00000100:00100000:3.0:1777502761.042374:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562100096 00000100:00100000:3.0:1777502761.042408:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562100224 00000100:00100000:3.0:1777502761.042412:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562100224 00000100:00100000:0.0:1777502761.042422:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16752 00000100:00100000:0.0:1777502761.042428:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6eb480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562100096:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502761.042457:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502761.042459:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16753 00000100:00100000:3.0:1777502761.042468:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6e9500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562100224:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502761.042479:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502761.042502:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6eb480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562100096:12345-0@lo:38:kworker.0 Request processed in 72us (289us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502761.042511:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16752 00000100:00100000:3.0:1777502761.042523:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6e9500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562100224:12345-0@lo:38:kworker.0 Request processed in 53us (235us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502761.042543:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16753 00000100:00100000:0.0:1777502761.042558:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6eb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562099968:0@lo:400:kworker.0 00000100:00080000:0.0:1777502761.042611:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502761.042616:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502761.042620:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6e8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562100096:0@lo:38:kworker.0 00000100:00080000:0.0:1777502761.042630:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502761.042634:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502761.042637:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6ebb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562100224:0@lo:38:kworker.0 00010000:00010000:0.0:1777502761.425728:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502762.449710:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502763.473803:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502764.497772:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502765.521816:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1777502766.161898:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562100352:0@lo:400:kworker.0 00000100:00080000:2.0:1777502766.161921:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502766.161929:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502766.161935:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502766.161943:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8888 00000100:00080000:2.0:1777502766.161948:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502766.161956:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502766.161986:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc560c8000 x1863837562100480/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502766.162040:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:2.0:1777502766.162189:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1777502766.162189:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562100352 00000100:00080000:2.0:1777502766.162197:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502766.162202:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1777502766.162202:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562100352 00000100:00100000:0.0:1777502766.162222:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562100480:0@lo:38:kworker.0 00000100:00100000:3.0:1777502766.162269:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 441 00000100:00100000:0.0:1777502766.162272:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502766.162282:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45ccdc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562100352:12345-0@lo:400:kworker.0 00000100:00080000:2.0:1777502766.162329:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8888 00000100:00080000:2.0:1777502766.162333:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502766.162338:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502766.162354:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562100480 00000100:00100000:0.0:1777502766.162361:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562100480 00000100:00100000:3.0:1777502766.162367:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45ccdc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562100352:12345-0@lo:400:kworker.0 Request processed in 89us (335us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502766.162405:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 441 00000100:00100000:0.0:1777502766.162437:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16754 00000100:00100000:0.0:1777502766.162448:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6ebb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562100480:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502766.162469:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502766.162548:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6ebb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562100480:12345-0@lo:38:kworker.0 Request processed in 98us (279us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502766.162560:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16754 00000100:00080000:0.0:1777502766.162681:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502766.162689:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502766.162696:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562100480:0@lo:38:kworker.0 00000100:00100000:0.0:1777502766.162764:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562100352:0@lo:400:kworker.0 00000100:00100000:3.0:1777502766.162807:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa35500 x1863837562100608/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502766.162853:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562100608:0@lo:38:kworker.0 00000100:00100000:3.0:1777502766.162895:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502766.162938:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562100608 00000100:00100000:3.0:1777502766.162941:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562100608 00000100:00100000:3.0:1777502766.162962:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16755 00000100:00100000:3.0:1777502766.162968:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4fa36d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562100608:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502766.162980:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502766.163117:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4fa36d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562100608:12345-0@lo:38:kworker.0 Request processed in 146us (221us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502766.163124:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16755 00000100:00080000:2.0:1777502766.163179:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502766.163186:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502766.163192:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562100608:0@lo:38:kworker.0 00010000:00010000:0.0:1777502766.545765:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502767.569754:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502768.593767:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502769.617773:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502770.641726:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502771.281803:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502771.281813:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502771.281819:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502771.281831:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8893 00000100:00080000:3.0:1777502771.281838:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502771.281847:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502771.281863:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa37100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562100736:0@lo:400:kworker.0 00000100:00100000:3.0:1777502771.281881:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa35c00 x1863837562100864/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502771.281908:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502771.281913:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502771.281917:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502771.281924:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8893 00000100:00080000:3.0:1777502771.281928:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1777502771.281951:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502771.282412:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562100864:0@lo:38:kworker.0 00000100:00100000:2.0:1777502771.282484:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502771.282902:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562100864 00000100:00100000:0.0:1777502771.282912:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562100864 00000100:00100000:0.0:1777502771.283026:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16756 00000100:00100000:0.0:1777502771.283038:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc560c9180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562100864:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502771.283061:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502771.283137:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc560c9180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562100864:12345-0@lo:38:kworker.0 Request processed in 101us (654us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502771.283158:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16756 00000100:00100000:0.0:1777502771.283193:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562100736 00000100:00100000:0.0:1777502771.283197:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562100736 00000100:00100000:0.0:1777502771.283252:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 442 00000100:00100000:0.0:1777502771.283265:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45cce850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562100736:12345-0@lo:400:kworker.0 00000100:00080000:3.0:1777502771.283282:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502771.283335:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45cce850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562100736:12345-0@lo:400:kworker.0 Request processed in 73us (1389us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502771.283359:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 442 00000100:00080000:0.0:1777502771.283469:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502771.283478:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502771.283483:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562100864:0@lo:38:kworker.0 00000100:00100000:0.0:1777502771.283537:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa37100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562100736:0@lo:400:kworker.0 00000100:00100000:2.0:1777502771.283551:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc560ca300 x1863837562100992/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502771.283626:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560ca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562100992:0@lo:38:kworker.0 00000100:00100000:0.0:1777502771.283664:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502771.283705:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562100992 00000100:00100000:3.0:1777502771.283710:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562100992 00000100:00100000:3.0:1777502771.283754:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16757 00000100:00100000:3.0:1777502771.283766:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6e8e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562100992:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502771.283780:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502771.283840:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6e8e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562100992:12345-0@lo:38:kworker.0 Request processed in 78us (177us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502771.283848:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16757 00000100:00080000:0.0:1777502771.283872:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502771.283879:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502771.283883:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560ca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562100992:0@lo:38:kworker.0 00010000:00010000:3.0:1777502771.666016:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502772.689727:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502773.713711:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502774.737818:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502775.761741:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502776.401715:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502776.401721:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502776.401725:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502776.401733:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8898 00000100:00080000:0.0:1777502776.401743:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502776.401763:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502776.401788:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd6e6ead80 x1863837562101248/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502776.401807:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502776.401810:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502776.401812:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502776.401817:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8898 00000100:00080000:0.0:1777502776.401819:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502776.401824:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502776.401841:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c6b800 x1863837562101376/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502776.401871:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6eb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562101120:0@lo:400:kworker.0 00000100:00100000:0.0:1777502776.401938:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502776.402015:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562101120 00000100:00100000:3.0:1777502776.402024:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562101120 00000100:00100000:3.0:1777502776.402073:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 443 00000100:00100000:3.0:1777502776.402084:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45cce050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562101120:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502776.402140:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45cce050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562101120:12345-0@lo:400:kworker.0 Request processed in 59us (203us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502776.402148:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 443 00000100:00100000:0.0:1777502776.402220:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6ead80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562101248:0@lo:38:kworker.0 00000100:00100000:0.0:1777502776.402270:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502776.402324:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562101248 00000100:00100000:3.0:1777502776.402328:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562101248 00000100:00100000:0.0:1777502776.402334:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562101376:0@lo:38:kworker.0 00000100:00100000:3.0:1777502776.402361:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16758 00000100:00100000:0.0:1777502776.402364:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502776.402367:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c68000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562101248:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502776.402382:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502776.402397:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562101376 00000100:00100000:2.0:1777502776.402404:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562101376 00000100:00100000:3.0:1777502776.402419:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c68000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562101248:12345-0@lo:38:kworker.0 Request processed in 51us (149us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502776.402427:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16758 00000100:00100000:2.0:1777502776.402438:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16759 00000100:00100000:0.0:1777502776.402440:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6eb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562101120:0@lo:400:kworker.0 00000100:00100000:2.0:1777502776.402446:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4772e300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562101376:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502776.402456:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00080000:0.0:1777502776.402483:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502776.402489:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502776.402494:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6ead80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562101248:0@lo:38:kworker.0 00000100:00100000:2.0:1777502776.402498:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4772e300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562101376:12345-0@lo:38:kworker.0 Request processed in 51us (134us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502776.402505:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16759 00000100:00080000:0.0:1777502776.402512:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502776.402518:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502776.402528:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562101376:0@lo:38:kworker.0 00010000:00010000:3.0:1777502776.785700:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502777.809713:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502778.833714:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502779.857797:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502780.881716:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502781.521770:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1777502781.521780:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562101504:0@lo:400:kworker.0 00000100:00080000:2.0:1777502781.521783:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502781.521787:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502781.521793:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8903 00000100:00080000:2.0:1777502781.521796:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502781.521801:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502781.521822:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc560c8e00 x1863837562101632/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502781.521842:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502781.521847:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502781.521849:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502781.521853:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8903 00000100:00080000:2.0:1777502781.521857:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1777502781.521861:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:2.0:1777502781.521864:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502781.521881:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc560cb800 x1863837562101760/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502781.521949:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562101632:0@lo:38:kworker.0 00000100:00100000:0.0:1777502781.521989:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502781.522018:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562101760:0@lo:38:kworker.0 00000100:00100000:0.0:1777502781.522060:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502781.522112:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562101504 00000100:00100000:0.0:1777502781.522119:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562101504 00000100:00100000:0.0:1777502781.522173:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 444 00000100:00100000:0.0:1777502781.522186:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45cccc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562101504:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502781.522262:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562101632 00000100:00100000:2.0:1777502781.522266:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562101632 00000100:00100000:2.0:1777502781.522289:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562101760 00000100:00100000:2.0:1777502781.522291:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562101760 00000100:00100000:2.0:1777502781.522311:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16760 00000100:00100000:2.0:1777502781.522317:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4772ed80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562101632:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502781.522331:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502781.522372:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4772ed80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562101632:12345-0@lo:38:kworker.0 Request processed in 54us (383us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502781.522381:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16761 00000100:00100000:2.0:1777502781.522389:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16760 00000100:00100000:3.0:1777502781.522391:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4772f100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562101760:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502781.522404:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502781.522449:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4772f100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562101760:12345-0@lo:38:kworker.0 Request processed in 58us (389us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502781.522458:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16761 00000100:00100000:0.0:1777502781.522658:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45cccc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562101504:12345-0@lo:400:kworker.0 Request processed in 474us (799us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502781.522667:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 444 00000100:00100000:2.0:1777502781.522694:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562101504:0@lo:400:kworker.0 00000100:00080000:0.0:1777502781.522725:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502781.522730:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502781.522734:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562101632:0@lo:38:kworker.0 00000100:00080000:0.0:1777502781.522748:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502781.522753:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502781.522757:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562101760:0@lo:38:kworker.0 00010000:00010000:0.0:1777502781.905743:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502782.929796:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502783.953759:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502784.977739:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502786.001739:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:02000400:2.0:1777502786.641678:0:221612:0:(genops.c:1873:obd_exports_barrier()) lustre-MDT0000 is waiting for obd_unlinked_exports more than 128 seconds. The obd refcount = 5. Is it stuck? 00000100:00080000:0.0:1777502786.641757:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502786.641763:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502786.641767:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502786.641775:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8908 00000100:00080000:0.0:1777502786.641780:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502786.641786:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502786.641810:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4772dc00 x1863837562102016/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502786.641828:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502786.641832:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502786.641834:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502786.641839:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8908 00000100:00080000:0.0:1777502786.641842:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502786.641847:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502786.641860:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4772c380 x1863837562102144/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502786.641914:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4772f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562101888:0@lo:400:kworker.0 00000100:00100000:0.0:1777502786.641989:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502786.642097:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4772dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562102016:0@lo:38:kworker.0 00000100:00100000:3.0:1777502786.642149:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502786.642213:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4772c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562102144:0@lo:38:kworker.0 00000100:00100000:3.0:1777502786.642245:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502786.642257:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562101888 00000100:00100000:0.0:1777502786.642265:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562101888 00000100:00100000:3.0:1777502786.642301:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562102016 00000100:00100000:3.0:1777502786.642307:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562102016 00000100:00100000:0.0:1777502786.642316:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 445 00000100:00100000:0.0:1777502786.642328:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45ccf450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562101888:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502786.642340:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562102144 00000100:00100000:3.0:1777502786.642344:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562102144 00000100:00100000:3.0:1777502786.642371:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16762 00000100:00100000:3.0:1777502786.642380:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4fa37480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562102016:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502786.642396:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502786.642444:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4fa37480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562102016:12345-0@lo:38:kworker.0 Request processed in 63us (294us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502786.642453:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16762 00000100:00100000:3.0:1777502786.642477:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16763 00000100:00100000:3.0:1777502786.642484:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4fa37100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562102144:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502786.642492:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502786.642525:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4fa37100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562102144:12345-0@lo:38:kworker.0 Request processed in 39us (279us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502786.642543:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16763 00000100:00080000:3.0:1777502786.642618:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502786.642625:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502786.642630:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4772dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562102016:0@lo:38:kworker.0 00000100:00080000:3.0:1777502786.642646:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502786.642651:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502786.642656:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4772c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562102144:0@lo:38:kworker.0 00000100:00100000:0.0:1777502786.642812:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45ccf450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562101888:12345-0@lo:400:kworker.0 Request processed in 486us (823us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502786.642822:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 445 00000100:00100000:3.0:1777502786.642830:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4772f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562101888:0@lo:400:kworker.0 00000020:02000400:2.0:1777502786.645135:0:221612:0:(genops.c:1840:print_export_data()) lustre-MDT0000: UNLINKED ffff8dfd66b1b800 0775d2ab-558e-41f1-82c0-14244d133390 192.168.202.27@tcp 1 (0 0 0) 1 0 0 0: 0000000000000000 8589949329 stale:0 00010000:00010000:0.0:1777502787.025727:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502788.049796:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502789.073729:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502790.097739:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502791.121692:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502791.761751:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502791.761759:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502791.761764:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502791.761773:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8913 00000100:00080000:2.0:1777502791.761778:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502791.761785:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502791.761792:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562102272:0@lo:400:kworker.0 00000100:00100000:2.0:1777502791.761810:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc560c9500 x1863837562102400/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502791.761829:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502791.761832:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502791.761834:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502791.761838:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8913 00000100:00080000:2.0:1777502791.761841:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502791.761846:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502791.761860:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc560cb800 x1863837562102528/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502791.761870:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502791.761927:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562102400:0@lo:38:kworker.0 00000100:00100000:0.0:1777502791.761946:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562102272 00000100:00100000:0.0:1777502791.761952:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562102272 00000100:00100000:2.0:1777502791.761967:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502791.761995:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 446 00000100:00100000:2.0:1777502791.761998:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562102528:0@lo:38:kworker.0 00000100:00100000:0.0:1777502791.762007:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45ccc050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562102272:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502791.762025:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502791.762071:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45ccc050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562102272:12345-0@lo:400:kworker.0 Request processed in 68us (203us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502791.762079:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 446 00000100:00100000:2.0:1777502791.762127:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562102400 00000100:00100000:0.0:1777502791.762134:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562102528 00000100:00100000:0.0:1777502791.762138:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562102528 00000100:00100000:2.0:1777502791.762142:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562102400 00000100:00100000:0.0:1777502791.762178:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16764 00000100:00100000:0.0:1777502791.762185:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc560ca300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562102528:12345-0@lo:38:kworker.0 00000100:00100000:2.0:1777502791.762190:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16765 00000100:00100000:2.0:1777502791.762196:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc560c8e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562102400:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502791.762201:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:2.0:1777502791.762205:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502791.762265:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc560ca300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562102528:12345-0@lo:38:kworker.0 Request processed in 77us (238us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502791.762269:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc560c8e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562102400:12345-0@lo:38:kworker.0 Request processed in 69us (300us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502791.762276:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16764 00000100:00100000:2.0:1777502791.762283:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16765 00000100:00080000:2.0:1777502791.762364:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502791.762369:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502791.762373:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562102400:0@lo:38:kworker.0 00000100:00080000:2.0:1777502791.762396:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502791.762400:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502791.762403:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562102528:0@lo:38:kworker.0 00000100:00100000:0.0:1777502791.762457:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562102272:0@lo:400:kworker.0 00010000:00010000:3.0:1777502792.145742:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502793.169742:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502794.193732:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502795.217796:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502796.241825:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502796.881779:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502796.881787:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502796.881792:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502796.881802:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8919 00000100:00080000:0.0:1777502796.881859:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502796.881870:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502796.881905:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4772c700 x1863837562102784/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502796.881936:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502796.881940:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502796.881945:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502796.881953:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8919 00000100:00080000:0.0:1777502796.881957:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502796.881963:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502796.881982:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4772e680 x1863837562102912/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502796.882022:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4772c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562102656:0@lo:400:kworker.0 00000100:00100000:0.0:1777502796.882092:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502796.882203:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4772c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562102784:0@lo:38:kworker.0 00000100:00100000:0.0:1777502796.882234:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502796.882254:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4772e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562102912:0@lo:38:kworker.0 00000100:00100000:0.0:1777502796.882280:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502796.882313:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562102656 00000100:00100000:0.0:1777502796.882319:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562102656 00000100:00100000:0.0:1777502796.882368:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 447 00000100:00100000:0.0:1777502796.882377:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd45ccc450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562102656:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502796.882425:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd45ccc450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562102656:12345-0@lo:400:kworker.0 Request processed in 50us (337us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502796.882434:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 447 00000100:00100000:0.0:1777502796.882502:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562102784 00000100:00100000:0.0:1777502796.882507:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562102784 00000100:00100000:0.0:1777502796.882535:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562102912 00000100:00100000:0.0:1777502796.882538:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562102912 00000100:00100000:0.0:1777502796.882609:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16766 00000100:00100000:0.0:1777502796.882616:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4772fb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562102784:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502796.882633:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502796.882708:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4772fb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562102784:12345-0@lo:38:kworker.0 Request processed in 86us (469us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502796.882722:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16766 00000100:00100000:0.0:1777502796.882760:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16767 00000100:00100000:0.0:1777502796.882765:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4772d500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562102912:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502796.882776:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502796.882823:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4772d500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562102912:12345-0@lo:38:kworker.0 Request processed in 54us (541us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502796.882832:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16767 00000100:00080000:0.0:1777502796.882878:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502796.882884:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502796.882888:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4772c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562102784:0@lo:38:kworker.0 00000100:00080000:0.0:1777502796.882904:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502796.882911:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502796.882913:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4772e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562102912:0@lo:38:kworker.0 00000100:00100000:0.0:1777502796.882962:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4772c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562102656:0@lo:400:kworker.0 00010000:00010000:3.0:1777502797.265764:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502798.289719:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502799.316138:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502800.339125:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502801.361802:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502802.001745:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502802.001753:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1777502802.001753:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562103040:0@lo:400:kworker.0 00000100:00080000:2.0:1777502802.001758:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502802.001765:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8924 00000100:00080000:2.0:1777502802.001770:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502802.001776:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502802.001799:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc560c9500 x1863837562103168/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502802.001817:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502802.001821:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502802.001824:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1777502802.001824:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:2.0:1777502802.001833:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8924 00000100:00080000:2.0:1777502802.001848:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502802.001853:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502802.001869:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc560c8380 x1863837562103296/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502802.001923:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562103040 00000100:00100000:3.0:1777502802.001930:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562103040 00000100:00100000:3.0:1777502802.001988:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 448 00000100:00100000:3.0:1777502802.002003:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562103040:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502802.002070:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562103168:0@lo:38:kworker.0 00000100:00100000:0.0:1777502802.002124:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502802.002154:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562103296:0@lo:38:kworker.0 00000100:00100000:0.0:1777502802.002182:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502802.002226:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562103168 00000100:00100000:2.0:1777502802.002232:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562103168 00000100:00100000:0.0:1777502802.002269:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562103296 00000100:00100000:0.0:1777502802.002274:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562103296 00000100:00100000:2.0:1777502802.002274:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16768 00000100:00100000:2.0:1777502802.002283:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4772c000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562103168:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502802.002299:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502802.002305:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16769 00000100:00100000:0.0:1777502802.002311:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4772e680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562103296:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502802.002321:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502802.002324:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562103040:12345-0@lo:400:kworker.0 Request processed in 324us (500us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502802.002335:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 448 00000100:00100000:2.0:1777502802.002356:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4772c000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562103168:12345-0@lo:38:kworker.0 Request processed in 71us (232us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502802.002365:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16768 00000100:00100000:0.0:1777502802.002368:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4772e680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562103296:12345-0@lo:38:kworker.0 Request processed in 54us (185us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502802.002375:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16769 00000100:00080000:2.0:1777502802.002409:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1777502802.002410:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562103040:0@lo:400:kworker.0 00000100:00080000:2.0:1777502802.002419:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502802.002424:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562103168:0@lo:38:kworker.0 00000100:00080000:2.0:1777502802.002438:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502802.002443:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502802.002447:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560c8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562103296:0@lo:38:kworker.0 00010000:00010000:0.0:1777502802.385820:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502803.409714:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502804.433737:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502805.457787:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502806.481794:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502807.121804:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502807.121811:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502807.121816:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502807.121826:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8929 00000100:00100000:2.0:1777502807.121832:0:3309:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:0.0:1777502807.121834:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502807.121844:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502807.121859:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4772c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562103424:0@lo:400:kworker.0 00000100:00100000:0.0:1777502807.121875:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4772f480 x1863837562103552/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502807.121897:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502807.121902:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502807.121904:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502807.121909:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8929 00000100:00100000:2.0:1777502807.121958:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:0.0:1777502807.122052:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502807.122063:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502807.122084:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4772f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562103552:0@lo:38:kworker.0 00000100:00100000:0.0:1777502807.122105:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4772d880 x1863837562103680/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502807.122126:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502807.122170:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4772d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562103680:0@lo:38:kworker.0 00000100:00100000:2.0:1777502807.122211:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502807.122275:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562103552 00000100:00100000:0.0:1777502807.122282:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562103552 00000100:00100000:0.0:1777502807.122330:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562103680 00000100:00100000:0.0:1777502807.122334:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562103680 00000100:00100000:2.0:1777502807.122342:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562103424 00000100:00100000:2.0:1777502807.122350:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562103424 00000100:00100000:0.0:1777502807.122361:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16770 00000100:00100000:0.0:1777502807.122367:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc560c8380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562103552:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502807.122385:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502807.122429:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 449 00000100:00100000:2.0:1777502807.122448:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1bc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562103424:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502807.122470:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc560c8380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562103552:12345-0@lo:38:kworker.0 Request processed in 98us (342us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502807.122481:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16770 00000100:00100000:2.0:1777502807.122525:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1bc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562103424:12345-0@lo:400:kworker.0 Request processed in 79us (563us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502807.122526:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16771 00000100:00100000:0.0:1777502807.122539:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc560c9500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562103680:12345-0@lo:38:kworker.0 00000100:00100000:2.0:1777502807.122546:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 449 00010000:02000000:0.0:1777502807.122558:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502807.122670:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc560c9500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562103680:12345-0@lo:38:kworker.0 Request processed in 130us (463us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502807.122678:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4772c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562103424:0@lo:400:kworker.0 00000100:00100000:0.0:1777502807.122687:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16771 00000100:00080000:3.0:1777502807.122988:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502807.122999:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502807.123018:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4772d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562103680:0@lo:38:kworker.0 00000100:00080000:3.0:1777502807.123046:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502807.123054:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502807.123060:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4772f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562103552:0@lo:38:kworker.0 00010000:00010000:0.0:1777502807.505776:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502808.529792:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502809.553760:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502810.577791:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502811.601851:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502812.241791:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502812.241800:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502812.241807:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502812.241823:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8934 00000100:00100000:0.0:1777502812.241830:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562103808:0@lo:400:kworker.0 00000100:00080000:2.0:1777502812.241831:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502812.241848:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502812.241883:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc560cb800 x1863837562103936/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502812.241906:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502812.241910:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502812.241913:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502812.241921:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8934 00000100:00080000:2.0:1777502812.241930:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1777502812.241937:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:2.0:1777502812.241942:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502812.241976:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc560c9c00 x1863837562104064/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502812.242044:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562103936:0@lo:38:kworker.0 00000100:00100000:2.0:1777502812.242141:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502812.242195:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562104064:0@lo:38:kworker.0 00000100:00100000:2.0:1777502812.242321:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502812.242385:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562103808 00000100:00100000:3.0:1777502812.242396:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562103808 00000100:00100000:0.0:1777502812.242491:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562103936 00000100:00100000:0.0:1777502812.242500:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562103936 00000100:00100000:0.0:1777502812.242537:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562104064 00000100:00100000:3.0:1777502812.242546:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 450 00000100:00100000:0.0:1777502812.242555:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562104064 00000100:00100000:0.0:1777502812.242620:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16772 00000100:00100000:0.0:1777502812.242628:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd509d3800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562103936:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502812.242643:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee19850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562103808:12345-0@lo:400:kworker.0 00010000:02000000:0.0:1777502812.242649:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502812.242884:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee19850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562103808:12345-0@lo:400:kworker.0 Request processed in 325us (960us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502812.242896:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 450 00000100:00100000:2.0:1777502812.243003:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560c8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562103808:0@lo:400:kworker.0 00000100:00100000:2.0:1777502812.243118:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16773 00000100:00100000:2.0:1777502812.243129:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc5ebf6a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562104064:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502812.243140:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502812.243204:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc5ebf6a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562104064:12345-0@lo:38:kworker.0 Request processed in 76us (883us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502812.243215:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16773 00000100:00080000:2.0:1777502812.243334:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502812.243344:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502812.243351:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560c9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562104064:0@lo:38:kworker.0 00000100:00100000:0.0:1777502812.246736:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd509d3800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562103936:12345-0@lo:38:kworker.0 Request processed in 4104us (4598us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502812.246743:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16772 00000100:00080000:2.0:1777502812.246781:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502812.246788:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502812.246793:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562103936:0@lo:38:kworker.0 00010000:00010000:0.0:1777502812.625767:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502813.649745:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502814.673822:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502815.697747:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502816.721755:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502817.361732:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502817.361736:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502817.361740:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502817.361746:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8939 00000100:00080000:0.0:1777502817.361750:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502817.361756:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502817.361774:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4772d880 x1863837562104320/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502817.361789:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502817.361792:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502817.361794:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502817.361797:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8939 00000100:00080000:0.0:1777502817.361800:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502817.361802:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502817.361813:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4772c700 x1863837562104448/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502817.361888:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:0.0:1777502817.361921:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4772f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562104192:0@lo:400:kworker.0 00000100:00100000:0.0:1777502817.361990:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502817.362068:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4772d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562104320:0@lo:38:kworker.0 00000100:00100000:0.0:1777502817.362097:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502817.362119:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4772c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562104448:0@lo:38:kworker.0 00000100:00100000:0.0:1777502817.362157:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502817.362209:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562104320 00000100:00100000:0.0:1777502817.362216:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562104320 00000100:00100000:0.0:1777502817.362247:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562104448 00000100:00100000:0.0:1777502817.362249:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562104448 00000100:00100000:0.0:1777502817.362272:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16774 00000100:00100000:0.0:1777502817.362278:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4772f800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562104320:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502817.362294:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502817.362346:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4772f800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562104320:12345-0@lo:38:kworker.0 Request processed in 66us (247us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502817.362354:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16774 00000100:00100000:0.0:1777502817.362378:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16775 00000100:00100000:0.0:1777502817.362383:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4772c380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562104448:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502817.362390:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502817.362428:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4772c380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562104448:12345-0@lo:38:kworker.0 Request processed in 42us (269us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502817.362436:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16775 00000100:00100000:0.0:1777502817.362472:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562104192 00000100:00100000:0.0:1777502817.362476:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562104192 00000100:00100000:0.0:1777502817.362511:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 451 00000100:00100000:0.0:1777502817.362526:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1a050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562104192:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502817.362568:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1a050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562104192:12345-0@lo:400:kworker.0 Request processed in 45us (581us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502817.362574:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 451 00000100:00080000:0.0:1777502817.362645:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502817.362652:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502817.362656:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4772f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562104192:0@lo:400:kworker.0 00000100:00100000:0.0:1777502817.362659:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4772d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562104320:0@lo:38:kworker.0 00000100:00080000:0.0:1777502817.362696:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502817.362702:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502817.362706:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4772c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562104448:0@lo:38:kworker.0 00010000:00010000:3.0:1777502817.745760:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502818.769723:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502819.793736:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502820.817737:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502821.841742:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502822.481887:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502822.481896:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502822.481901:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502822.481914:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8944 00000100:00080000:2.0:1777502822.481922:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502822.481933:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502822.481981:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc5ebf4700 x1863837562104704/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502822.482035:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502822.482040:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502822.482045:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502822.482052:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8944 00000100:00080000:2.0:1777502822.482057:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502822.482065:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502822.482096:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc5ebf7800 x1863837562104832/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502822.482152:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562104576:0@lo:400:kworker.0 00000100:00100000:2.0:1777502822.482272:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502822.482306:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562104704:0@lo:38:kworker.0 00000100:00100000:0.0:1777502822.482385:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502822.482425:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562104832:0@lo:38:kworker.0 00000100:00100000:2.0:1777502822.482457:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562104576 00000100:00100000:2.0:1777502822.482467:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562104576 00000100:00100000:2.0:1777502822.482558:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 452 00000100:00100000:2.0:1777502822.482575:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee19050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562104576:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502822.482634:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502822.482741:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562104704 00000100:00100000:2.0:1777502822.482749:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562104704 00000100:00100000:2.0:1777502822.482793:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562104832 00000100:00100000:2.0:1777502822.482798:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562104832 00000100:00100000:2.0:1777502822.482847:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16776 00000100:00100000:2.0:1777502822.482867:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4772c700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562104704:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502822.482892:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502822.483057:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16777 00000100:00100000:2.0:1777502822.483068:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4772d880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562104832:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502822.483081:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502822.483179:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee19050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562104576:12345-0@lo:400:kworker.0 Request processed in 606us (910us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502822.483190:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 452 00000100:00100000:2.0:1777502822.483253:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4772d880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562104832:12345-0@lo:38:kworker.0 Request processed in 185us (619us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502822.483272:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16777 00000100:00100000:3.0:1777502822.483308:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4772c700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562104704:12345-0@lo:38:kworker.0 Request processed in 449us (927us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502822.483331:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16776 00000100:00100000:2.0:1777502822.483370:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562104576:0@lo:400:kworker.0 00000100:00080000:2.0:1777502822.483451:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502822.483460:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502822.483469:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562104704:0@lo:38:kworker.0 00000100:00080000:2.0:1777502822.483493:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502822.483503:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502822.483509:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562104832:0@lo:38:kworker.0 00010000:00010000:0.0:1777502822.865890:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502823.889772:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502824.913711:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502825.937733:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502826.963072:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502827.601828:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502827.601837:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502827.601845:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502827.601859:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8949 00000100:00080000:3.0:1777502827.601865:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502827.601875:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502827.601914:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa35500 x1863837562105088/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502827.601935:0:3309:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:3.0:1777502827.601942:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502827.601945:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502827.601948:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502827.601953:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8949 00000100:00080000:3.0:1777502827.601957:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502827.601965:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502827.601968:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562104960:0@lo:400:kworker.0 00000100:00100000:3.0:1777502827.601989:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc493f8700 x1863837562105216/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502827.602079:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502827.602204:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562105088:0@lo:38:kworker.0 00000100:00100000:3.0:1777502827.602261:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502827.602300:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562105216:0@lo:38:kworker.0 00000100:00100000:2.0:1777502827.602313:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562105088 00000100:00100000:2.0:1777502827.602325:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562105088 00000100:00100000:3.0:1777502827.602337:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502827.602356:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562104960 00000100:00100000:0.0:1777502827.602370:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562104960 00000100:00100000:2.0:1777502827.602381:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562105216 00000100:00100000:2.0:1777502827.602386:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562105216 00000100:00100000:3.0:1777502827.602415:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16778 00000100:00100000:3.0:1777502827.602430:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493fa680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562105088:12345-0@lo:38:kworker.0 00000100:00100000:2.0:1777502827.602437:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16779 00000100:00100000:2.0:1777502827.602445:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493fb800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562105216:12345-0@lo:38:kworker.0 00000100:00100000:0.0:1777502827.602454:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 453 00010000:02000000:2.0:1777502827.602472:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502827.602474:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1a450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562104960:12345-0@lo:400:kworker.0 00010000:02000000:3.0:1777502827.602478:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502827.602554:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493fa680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562105088:12345-0@lo:38:kworker.0 Request processed in 127us (295us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502827.602567:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16778 00000100:00100000:2.0:1777502827.602575:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493fb800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562105216:12345-0@lo:38:kworker.0 Request processed in 126us (236us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502827.602622:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16779 00000100:00100000:0.0:1777502827.602655:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1a450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562104960:12345-0@lo:400:kworker.0 Request processed in 190us (577us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502827.602700:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562104960:0@lo:400:kworker.0 00000100:00100000:0.0:1777502827.602710:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 453 00000100:00080000:3.0:1777502827.602751:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502827.602763:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502827.602771:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562105088:0@lo:38:kworker.0 00000100:00080000:3.0:1777502827.602796:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502827.602802:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502827.602820:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562105216:0@lo:38:kworker.0 00010000:00010000:0.0:1777502827.985759:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502829.009817:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502830.033770:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502831.057805:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502832.081760:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502832.721729:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502832.721736:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502832.721740:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502832.721749:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8954 00000100:00080000:3.0:1777502832.721754:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502832.721775:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502832.721805:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c53480 x1863837562105472/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502832.721826:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1777502832.721828:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00080000:3.0:1777502832.721830:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502832.721834:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502832.721839:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8954 00000100:00080000:3.0:1777502832.721842:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502832.721846:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502832.721852:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562105344:0@lo:400:kworker.0 00000100:00100000:3.0:1777502832.721876:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c50e00 x1863837562105600/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502832.721904:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c53480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562105472:0@lo:38:kworker.0 00000100:00100000:0.0:1777502832.721926:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502832.721952:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502832.721981:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c50e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562105600:0@lo:38:kworker.0 00000100:00100000:3.0:1777502832.722011:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502832.722044:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562105344 00000100:00100000:0.0:1777502832.722050:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562105344 00000100:00100000:3.0:1777502832.722096:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562105472 00000100:00100000:3.0:1777502832.722101:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562105472 00000100:00100000:0.0:1777502832.722102:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 454 00000100:00100000:0.0:1777502832.722115:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee18450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562105344:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502832.722131:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562105600 00000100:00100000:3.0:1777502832.722135:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562105600 00000100:00100000:3.0:1777502832.722162:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16780 00000100:00100000:3.0:1777502832.722170:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c51880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562105472:12345-0@lo:38:kworker.0 00000100:00100000:0.0:1777502832.722175:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee18450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562105344:12345-0@lo:400:kworker.0 Request processed in 61us (247us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502832.722183:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 454 00010000:02000000:3.0:1777502832.722188:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502832.722236:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c51880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562105472:12345-0@lo:38:kworker.0 Request processed in 66us (285us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502832.722244:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16780 00000100:00100000:3.0:1777502832.722266:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16781 00000100:00100000:3.0:1777502832.722271:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c50700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562105600:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502832.722280:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502832.722319:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c50700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562105600:12345-0@lo:38:kworker.0 Request processed in 46us (308us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502832.722327:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16781 00000100:00080000:3.0:1777502832.722386:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502832.722392:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502832.722396:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c53480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562105472:0@lo:38:kworker.0 00000100:00080000:3.0:1777502832.722412:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502832.722491:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502832.722496:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c50e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562105600:0@lo:38:kworker.0 00000100:00100000:0.0:1777502832.722541:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562105344:0@lo:400:kworker.0 00010000:00010000:0.0:1777502833.105734:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502834.129760:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502835.153726:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502836.177735:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502837.201705:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502837.841766:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502837.841771:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1777502837.841777:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562105728:0@lo:400:kworker.0 00000100:00080000:2.0:1777502837.841786:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502837.841791:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8959 00000100:00080000:2.0:1777502837.841795:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502837.841801:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502837.841820:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc5ebf4700 x1863837562105856/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502837.841836:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502837.841838:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502837.841840:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502837.841844:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8959 00000100:00080000:2.0:1777502837.841846:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502837.841850:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502837.841860:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc5ebf7b80 x1863837562105984/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502837.841913:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562105856:0@lo:38:kworker.0 00000100:00100000:0.0:1777502837.842965:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502837.842991:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502837.843008:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562105728 00000100:00100000:0.0:1777502837.843014:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562105728 00000100:00100000:2.0:1777502837.843041:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562105984:0@lo:38:kworker.0 00000100:00100000:2.0:1777502837.843063:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502837.843067:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 455 00000100:00100000:0.0:1777502837.843078:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee18850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562105728:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502837.843106:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562105856 00000100:00100000:2.0:1777502837.843110:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562105856 00000100:00100000:2.0:1777502837.843140:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562105984 00000100:00100000:2.0:1777502837.843143:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562105984 00000100:00100000:0.0:1777502837.843157:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee18850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562105728:12345-0@lo:400:kworker.0 Request processed in 79us (191us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502837.843167:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16782 00000100:00100000:0.0:1777502837.843168:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 455 00000100:00100000:2.0:1777502837.843176:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc5ebf4e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562105856:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502837.843190:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502837.843220:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16783 00000100:00100000:0.0:1777502837.843226:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc5ebf6300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562105984:12345-0@lo:38:kworker.0 00000100:00100000:2.0:1777502837.843230:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc5ebf4e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562105856:12345-0@lo:38:kworker.0 Request processed in 53us (239us total) trans 0 rc -19/-19 00010000:02000000:0.0:1777502837.843239:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502837.843245:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16782 00000100:00100000:0.0:1777502837.843275:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc5ebf6300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562105984:12345-0@lo:38:kworker.0 Request processed in 47us (212us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502837.843282:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16783 00000100:00080000:2.0:1777502837.843300:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502837.843307:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502837.843314:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562105856:0@lo:38:kworker.0 00000100:00100000:0.0:1777502837.843334:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562105728:0@lo:400:kworker.0 00000100:00080000:2.0:1777502837.843334:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502837.843343:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502837.843348:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562105984:0@lo:38:kworker.0 00010000:00010000:3.0:1777502838.225730:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502839.249724:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502840.273735:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502841.297730:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502842.321756:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502842.961721:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502842.961725:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502842.961729:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502842.961735:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8965 00000100:00080000:2.0:1777502842.961739:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502842.961744:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502842.961763:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc5ebf4700 x1863837562106240/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502842.961776:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502842.961778:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502842.961780:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502842.961783:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8965 00000100:00080000:2.0:1777502842.961785:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502842.961789:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502842.961800:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc5ebf5880 x1863837562106368/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502842.961826:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562106112:0@lo:400:kworker.0 00000100:00100000:2.0:1777502842.961925:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502842.962001:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562106240:0@lo:38:kworker.0 00000100:00100000:2.0:1777502842.962037:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502842.962060:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562106368:0@lo:38:kworker.0 00000100:00100000:2.0:1777502842.962101:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502842.962148:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562106112 00000100:00100000:2.0:1777502842.962153:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562106112 00000100:00100000:2.0:1777502842.962191:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 456 00000100:00100000:2.0:1777502842.962200:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1c850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562106112:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502842.962240:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1c850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562106112:12345-0@lo:400:kworker.0 Request processed in 43us (317us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502842.962246:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 456 00000100:00100000:2.0:1777502842.962280:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562106240 00000100:00100000:2.0:1777502842.962283:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562106240 00000100:00100000:2.0:1777502842.962302:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562106368 00000100:00100000:2.0:1777502842.962304:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562106368 00000100:00100000:2.0:1777502842.962325:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16784 00000100:00100000:2.0:1777502842.962329:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc5ebf7100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562106240:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502842.962341:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502842.962373:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc5ebf7100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562106240:12345-0@lo:38:kworker.0 Request processed in 42us (337us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502842.962378:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16784 00000100:00100000:2.0:1777502842.962396:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16785 00000100:00100000:2.0:1777502842.962401:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc5ebf6d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562106368:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502842.962407:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502842.962431:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc5ebf6d80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562106368:12345-0@lo:38:kworker.0 Request processed in 29us (330us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502842.962441:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16785 00000100:00080000:2.0:1777502842.962499:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502842.962503:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502842.962507:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562106240:0@lo:38:kworker.0 00000100:00080000:2.0:1777502842.962519:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502842.962521:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502842.962525:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562106368:0@lo:38:kworker.0 00000100:00100000:3.0:1777502842.962536:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562106112:0@lo:400:kworker.0 00010000:00010000:0.0:1777502843.345755:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502844.369779:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502845.393734:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502846.417722:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502847.441724:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502848.081765:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502848.081773:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502848.081777:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502848.081786:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8970 00000100:00080000:3.0:1777502848.081790:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502848.081797:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502848.081822:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c53480 x1863837562106624/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502848.081842:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502848.081846:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502848.081848:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502848.081853:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8970 00000100:00080000:3.0:1777502848.081856:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502848.081861:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502848.081876:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c51f80 x1863837562106752/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502848.081903:0:3309:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1777502848.081921:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c50e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562106496:0@lo:400:kworker.0 00000100:00100000:3.0:1777502848.082005:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502848.082085:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c53480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562106624:0@lo:38:kworker.0 00000100:00100000:3.0:1777502848.082121:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502848.082159:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c51f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562106752:0@lo:38:kworker.0 00000100:00100000:2.0:1777502848.082160:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562106624 00000100:00100000:2.0:1777502848.082169:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562106624 00000100:00100000:3.0:1777502848.082194:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502848.082208:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562106752 00000100:00100000:2.0:1777502848.082213:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562106752 00000100:00100000:2.0:1777502848.082242:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16786 00000100:00100000:3.0:1777502848.082242:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562106496 00000100:00100000:3.0:1777502848.082247:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562106496 00000100:00100000:2.0:1777502848.082253:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c53800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562106624:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502848.082268:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502848.082294:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 457 00000100:00100000:3.0:1777502848.082306:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1f050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562106496:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502848.082367:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1f050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562106496:12345-0@lo:400:kworker.0 Request processed in 63us (362us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502848.082375:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 457 00000100:00100000:2.0:1777502848.082391:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c53800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562106624:12345-0@lo:38:kworker.0 Request processed in 139us (270us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502848.082397:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16786 00000100:00100000:2.0:1777502848.082416:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16787 00000100:00100000:2.0:1777502848.082421:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c50a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562106752:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502848.082427:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502848.082441:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c50e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562106496:0@lo:400:kworker.0 00000100:00100000:2.0:1777502848.082453:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c50a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562106752:12345-0@lo:38:kworker.0 Request processed in 31us (262us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502848.082459:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16787 00000100:00080000:2.0:1777502848.082509:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502848.082513:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502848.082516:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c53480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562106624:0@lo:38:kworker.0 00000100:00080000:2.0:1777502848.082537:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502848.082541:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502848.082544:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c51f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562106752:0@lo:38:kworker.0 00010000:00010000:0.0:1777502848.465737:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502849.489773:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502850.513793:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502851.537823:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502852.561748:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502853.201780:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502853.201788:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502853.201793:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1777502853.201795:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00080000:3.0:1777502853.201807:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8975 00000100:00080000:3.0:1777502853.201813:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502853.201823:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502853.201825:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c50e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562106880:0@lo:400:kworker.0 00000100:00100000:3.0:1777502853.201852:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c53100 x1863837562107008/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502853.201878:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502853.201882:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1777502853.201932:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:3.0:1777502853.201997:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502853.202012:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8975 00000100:00080000:3.0:1777502853.202020:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502853.202031:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502853.202079:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c51500 x1863837562107136/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502853.202150:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c53100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562107008:0@lo:38:kworker.0 00000100:00100000:3.0:1777502853.202220:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502853.202273:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c51500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562107136:0@lo:38:kworker.0 00000100:00100000:3.0:1777502853.202309:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502853.202405:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562106880 00000100:00100000:3.0:1777502853.202414:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562106880 00000100:00100000:2.0:1777502853.202452:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562107008 00000100:00100000:2.0:1777502853.202461:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562107008 00000100:00100000:3.0:1777502853.202496:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 458 00000100:00100000:2.0:1777502853.202510:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562107136 00000100:00100000:3.0:1777502853.202512:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1dc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562106880:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502853.202516:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562107136 00000100:00100000:3.0:1777502853.202642:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1dc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562106880:12345-0@lo:400:kworker.0 Request processed in 133us (710us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502853.202645:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16788 00000100:00100000:3.0:1777502853.202670:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 458 00000100:00100000:2.0:1777502853.202671:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c52680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562107008:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502853.202702:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502853.202769:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c50e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562106880:0@lo:400:kworker.0 00000100:00100000:2.0:1777502853.202806:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16789 00000100:00100000:2.0:1777502853.202813:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c51f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562107136:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502853.202824:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502853.202900:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c51f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562107136:12345-0@lo:38:kworker.0 Request processed in 83us (597us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502853.202910:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16789 00000100:00100000:3.0:1777502853.202932:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c52680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562107008:12345-0@lo:38:kworker.0 Request processed in 269us (711us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502853.202942:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16788 00000100:00080000:3.0:1777502853.203025:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502853.203036:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502853.203044:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c53100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562107008:0@lo:38:kworker.0 00000100:00080000:3.0:1777502853.203074:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502853.203083:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502853.203117:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c51500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562107136:0@lo:38:kworker.0 00010000:00010000:0.0:1777502853.585757:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502854.609795:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502855.633832:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502856.657777:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502857.681790:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502858.321863:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502858.321869:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502858.321874:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502858.321883:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8980 00000100:00080000:3.0:1777502858.321889:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1777502858.321892:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c51500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562107264:0@lo:400:kworker.0 00000100:00080000:3.0:1777502858.321899:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502858.321925:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c53100 x1863837562107392/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502858.321943:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502858.321947:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502858.321949:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502858.321964:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8980 00000100:00100000:0.0:1777502858.321965:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:3.0:1777502858.321972:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502858.321978:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502858.321991:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c53480 x1863837562107520/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502858.322027:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c53100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562107392:0@lo:38:kworker.0 00000100:00100000:0.0:1777502858.322092:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562107264 00000100:00100000:3.0:1777502858.322092:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502858.322103:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562107264 00000100:00100000:3.0:1777502858.322138:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c53480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562107520:0@lo:38:kworker.0 00000100:00100000:0.0:1777502858.322161:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 459 00000100:00100000:3.0:1777502858.322174:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502858.322183:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1d450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562107264:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502858.322246:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562107392 00000100:00100000:3.0:1777502858.322254:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562107392 00000100:00100000:0.0:1777502858.322260:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1d450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562107264:12345-0@lo:400:kworker.0 Request processed in 88us (296us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502858.322267:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 459 00000100:00100000:3.0:1777502858.322302:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562107520 00000100:00100000:3.0:1777502858.322323:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562107520 00000100:00100000:0.0:1777502858.322356:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c51500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562107264:0@lo:400:kworker.0 00000100:00100000:3.0:1777502858.322361:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16790 00000100:00100000:3.0:1777502858.322375:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c50e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562107392:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502858.322393:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502858.322435:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16791 00000100:00100000:0.0:1777502858.322444:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4319ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562107520:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502858.322450:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c50e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562107392:12345-0@lo:38:kworker.0 Request processed in 71us (358us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502858.322467:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16790 00010000:02000000:0.0:1777502858.322475:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502858.322553:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4319ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562107520:12345-0@lo:38:kworker.0 Request processed in 107us (381us total) trans 0 rc -19/-19 00000100:00080000:3.0:1777502858.322560:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1777502858.322572:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16791 00000100:00080000:3.0:1777502858.322606:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502858.322611:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c53100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562107392:0@lo:38:kworker.0 00000100:00080000:3.0:1777502858.322637:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502858.322642:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502858.322648:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c53480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562107520:0@lo:38:kworker.0 00010000:00010000:3.0:1777502858.705776:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502859.729841:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502860.753744:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502861.777749:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502862.801717:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502863.441742:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502863.441749:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1777502863.441750:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562107648:0@lo:400:kworker.0 00000100:00080000:2.0:1777502863.441753:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502863.441758:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8985 00000100:00080000:2.0:1777502863.441762:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502863.441768:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502863.441790:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc5ebf4700 x1863837562107776/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502863.441808:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1777502863.441818:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:2.0:1777502863.441825:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502863.441827:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502863.441832:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8985 00000100:00080000:2.0:1777502863.441834:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502863.441839:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502863.441852:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc5ebf7b80 x1863837562107904/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502863.441885:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562107776:0@lo:38:kworker.0 00000100:00100000:2.0:1777502863.441902:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562107648 00000100:00100000:2.0:1777502863.441909:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562107648 00000100:00100000:3.0:1777502863.441921:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502863.441948:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562107904:0@lo:38:kworker.0 00000100:00100000:2.0:1777502863.441956:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 460 00000100:00100000:2.0:1777502863.441965:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1f450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562107648:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502863.441973:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502863.442013:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562107776 00000100:00100000:3.0:1777502863.442017:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562107776 00000100:00100000:2.0:1777502863.442018:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1f450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562107648:12345-0@lo:400:kworker.0 Request processed in 54us (200us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502863.442025:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 460 00000100:00100000:3.0:1777502863.442040:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562107904 00000100:00100000:3.0:1777502863.442043:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562107904 00000100:00100000:3.0:1777502863.442062:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16792 00000100:00100000:3.0:1777502863.442068:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4319d500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562107776:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502863.442125:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502863.442165:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4319d500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562107776:12345-0@lo:38:kworker.0 Request processed in 95us (242us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502863.442174:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16792 00000100:00100000:3.0:1777502863.442192:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16793 00000100:00100000:3.0:1777502863.442197:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4319f800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562107904:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502863.442204:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502863.442233:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4319f800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562107904:12345-0@lo:38:kworker.0 Request processed in 34us (258us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502863.442239:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16793 00000100:00080000:3.0:1777502863.442314:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502863.442320:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502863.442324:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562107776:0@lo:38:kworker.0 00000100:00080000:3.0:1777502863.442340:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502863.442344:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502863.442349:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562107904:0@lo:38:kworker.0 00000100:00100000:3.0:1777502863.442392:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562107648:0@lo:400:kworker.0 00010000:00010000:0.0:1777502863.825738:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502864.849826:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502865.873743:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502866.897736:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502867.921731:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502868.562397:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502868.562403:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502868.562408:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502868.562427:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8990 00000100:00080000:3.0:1777502868.562433:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502868.562440:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502868.562463:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4319e300 x1863837562108160/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502868.562492:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502868.562496:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502868.562498:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502868.562504:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8990 00000100:00080000:3.0:1777502868.562507:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502868.562511:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502868.562537:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4319df80 x1863837562108288/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502868.562568:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562108032:0@lo:400:kworker.0 00000100:00100000:0.0:1777502868.562625:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562108160:0@lo:38:kworker.0 00000100:00100000:3.0:1777502868.562643:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502868.562668:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502868.562729:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562108288:0@lo:38:kworker.0 00000100:00100000:0.0:1777502868.562754:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502868.562760:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562108032 00000100:00100000:3.0:1777502868.562768:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562108032 00000100:00100000:0.0:1777502868.562790:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562108160 00000100:00100000:0.0:1777502868.562794:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562108160 00000100:00100000:3.0:1777502868.562814:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 461 00000100:00100000:0.0:1777502868.562819:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562108288 00000100:00100000:0.0:1777502868.562822:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562108288 00000100:00100000:3.0:1777502868.562825:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1cc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562108032:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502868.562841:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16794 00000100:00100000:0.0:1777502868.562848:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4772dc00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562108160:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502868.562865:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502868.562871:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1cc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562108032:12345-0@lo:400:kworker.0 Request processed in 51us (229us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502868.562877:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 461 00000100:00100000:0.0:1777502868.562913:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4772dc00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562108160:12345-0@lo:38:kworker.0 Request processed in 63us (246us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502868.562922:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16794 00000100:00100000:3.0:1777502868.562928:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562108032:0@lo:400:kworker.0 00000100:00100000:0.0:1777502868.562945:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16795 00000100:00100000:0.0:1777502868.562951:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4772f480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562108288:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502868.562960:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502868.562990:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4772f480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562108288:12345-0@lo:38:kworker.0 Request processed in 39us (236us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502868.562995:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16795 00000100:00080000:0.0:1777502868.563023:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502868.563027:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502868.563029:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562108160:0@lo:38:kworker.0 00000100:00080000:0.0:1777502868.563038:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502868.563041:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502868.563044:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562108288:0@lo:38:kworker.0 00010000:00010000:0.0:1777502868.945732:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502869.969772:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502870.993875:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502872.017828:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502873.041728:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502873.681755:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502873.681761:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502873.681766:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502873.681776:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 8995 00000100:00080000:3.0:1777502873.681781:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1777502873.681782:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562108416:0@lo:400:kworker.0 00000100:00080000:3.0:1777502873.681835:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502873.681852:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502873.681861:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4319df80 x1863837562108544/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502873.681883:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502873.681887:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502873.681890:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502873.681895:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 8995 00000100:00080000:3.0:1777502873.681898:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502873.681903:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502873.681920:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4319e300 x1863837562108672/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502873.681922:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562108416 00000100:00100000:0.0:1777502873.681930:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562108416 00000100:00100000:0.0:1777502873.681970:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 462 00000100:00100000:3.0:1777502873.681973:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562108544:0@lo:38:kworker.0 00000100:00100000:0.0:1777502873.681980:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1e850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562108416:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502873.682019:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502873.682030:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1e850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562108416:12345-0@lo:400:kworker.0 Request processed in 51us (178us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502873.682038:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 462 00000100:00100000:3.0:1777502873.682055:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562108672:0@lo:38:kworker.0 00000100:00100000:0.0:1777502873.682105:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562108416:0@lo:400:kworker.0 00000100:00100000:2.0:1777502873.682683:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562108544 00000100:00100000:2.0:1777502873.682689:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562108544 00000100:00100000:0.0:1777502873.682784:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16796 00000100:00100000:0.0:1777502873.682792:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76df80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562108544:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502873.682847:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502873.683002:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502873.683034:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76df80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562108544:12345-0@lo:38:kworker.0 Request processed in 240us (1016us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502873.683041:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562108672 00000100:00100000:0.0:1777502873.683045:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16796 00000100:00100000:3.0:1777502873.683048:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562108672 00000100:00100000:0.0:1777502873.683073:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16797 00000100:00100000:0.0:1777502873.683080:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76c380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562108672:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502873.683089:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00080000:3.0:1777502873.683111:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502873.683118:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502873.683124:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562108544:0@lo:38:kworker.0 00000100:00100000:0.0:1777502873.683131:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76c380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562108672:12345-0@lo:38:kworker.0 Request processed in 48us (128us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502873.683139:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16797 00000100:00080000:3.0:1777502873.683141:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502873.683147:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502873.683151:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562108672:0@lo:38:kworker.0 00010000:00010000:0.0:1777502874.065732:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502875.089741:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502876.113719:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502877.137772:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502878.161753:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502878.801702:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502878.801709:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502878.801713:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502878.801721:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9000 00000100:00080000:3.0:1777502878.801726:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502878.801733:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502878.801755:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76c000 x1863837562108928/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502878.801788:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502878.801792:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502878.801794:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502878.801799:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9000 00000100:00080000:3.0:1777502878.801801:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502878.801806:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502878.801820:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76e680 x1863837562109056/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502878.801850:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562108800:0@lo:400:kworker.0 00000100:00100000:3.0:1777502878.801912:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502878.801989:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562108800 00000100:00100000:3.0:1777502878.801995:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562108800 00000100:00100000:3.0:1777502878.802042:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 463 00000100:00100000:3.0:1777502878.802053:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1f850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562108800:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502878.802108:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1f850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562108800:12345-0@lo:400:kworker.0 Request processed in 58us (197us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502878.802117:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 463 00000100:00100000:3.0:1777502878.802227:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562108800:0@lo:400:kworker.0 00000100:00100000:3.0:1777502878.802258:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562108928:0@lo:38:kworker.0 00000100:00100000:3.0:1777502878.802291:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502878.802334:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562109056:0@lo:38:kworker.0 00000100:00100000:3.0:1777502878.802364:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502878.802407:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562108928 00000100:00100000:2.0:1777502878.802412:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562108928 00000100:00100000:0.0:1777502878.802415:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562109056 00000100:00100000:0.0:1777502878.802421:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562109056 00000100:00100000:2.0:1777502878.802450:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16798 00000100:00100000:2.0:1777502878.802457:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76fb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562108928:12345-0@lo:38:kworker.0 00000100:00100000:0.0:1777502878.802461:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16799 00000100:00100000:0.0:1777502878.802470:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76f800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562109056:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502878.802477:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:0.0:1777502878.802482:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502878.802523:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76fb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562108928:12345-0@lo:38:kworker.0 Request processed in 64us (232us total) trans 0 rc -19/-19 00000100:00080000:3.0:1777502878.802529:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1777502878.802535:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76f800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562109056:12345-0@lo:38:kworker.0 Request processed in 63us (170us total) trans 0 rc -19/-19 00000100:00080000:3.0:1777502878.802537:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502878.802544:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16798 00000100:00100000:3.0:1777502878.802544:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562108928:0@lo:38:kworker.0 00000100:00100000:0.0:1777502878.802552:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16799 00000100:00080000:3.0:1777502878.802559:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502878.802565:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502878.802568:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562109056:0@lo:38:kworker.0 00010000:00010000:3.0:1777502879.185732:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502880.209797:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502881.233772:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502882.257757:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502883.281750:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502883.921752:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502883.921757:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502883.921761:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502883.921768:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9006 00000100:00080000:0.0:1777502883.921772:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502883.921778:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502883.921801:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4319df80 x1863837562109312/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502883.921818:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502883.921822:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502883.921824:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502883.921830:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9006 00000100:00080000:0.0:1777502883.921833:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502883.921837:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502883.921854:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4319c380 x1863837562109440/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502883.921939:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562109184:0@lo:400:kworker.0 00000100:00100000:0.0:1777502883.921997:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502883.922077:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562109184 00000100:00100000:0.0:1777502883.922084:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562109184 00000100:00100000:0.0:1777502883.922149:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 464 00000100:00100000:0.0:1777502883.922162:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1c050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562109184:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502883.922215:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1c050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562109184:12345-0@lo:400:kworker.0 Request processed in 57us (218us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502883.922222:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 464 00000100:00100000:0.0:1777502883.922296:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562109184:0@lo:400:kworker.0 00000100:00100000:3.0:1777502883.922306:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562109312:0@lo:38:kworker.0 00000100:00100000:3.0:1777502883.922402:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502883.922427:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562109440:0@lo:38:kworker.0 00000100:00100000:3.0:1777502883.922482:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502883.922514:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562109312 00000100:00100000:0.0:1777502883.922520:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562109312 00000100:00100000:3.0:1777502883.922548:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562109440 00000100:00100000:3.0:1777502883.922553:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562109440 00000100:00100000:0.0:1777502883.922554:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16800 00000100:00100000:0.0:1777502883.922563:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76e680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562109312:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502883.922599:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16801 00010000:02000000:0.0:1777502883.922601:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502883.922608:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76c000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562109440:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502883.922618:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502883.922640:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76e680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562109312:12345-0@lo:38:kworker.0 Request processed in 76us (238us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502883.922646:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16800 00000100:00100000:3.0:1777502883.922661:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76c000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562109440:12345-0@lo:38:kworker.0 Request processed in 52us (178us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502883.922668:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16801 00000100:00080000:2.0:1777502883.922744:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502883.922750:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502883.922754:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562109312:0@lo:38:kworker.0 00000100:00080000:2.0:1777502883.922774:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502883.922778:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502883.922781:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562109440:0@lo:38:kworker.0 00010000:00010000:0.0:1777502884.305741:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502885.329745:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502886.353716:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502887.377742:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502888.401720:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502889.041714:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502889.041720:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502889.041723:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502889.041732:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9011 00000100:00080000:0.0:1777502889.041748:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502889.041755:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502889.041777:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4319c380 x1863837562109696/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502889.041795:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502889.041799:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502889.041801:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502889.041805:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9011 00000100:00080000:0.0:1777502889.041808:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502889.041812:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502889.041826:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4319df80 x1863837562109824/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502889.041853:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562109568:0@lo:400:kworker.0 00000100:00100000:0.0:1777502889.041914:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502889.041997:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562109568 00000100:00100000:0.0:1777502889.042006:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562109568 00000100:00100000:0.0:1777502889.042043:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 465 00000100:00100000:0.0:1777502889.042052:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1ec50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562109568:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502889.042098:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1ec50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562109568:12345-0@lo:400:kworker.0 Request processed in 47us (185us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502889.042104:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562109696:0@lo:38:kworker.0 00000100:00100000:0.0:1777502889.042107:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 465 00000100:00100000:2.0:1777502889.042153:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502889.042156:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562109568:0@lo:400:kworker.0 00000100:00100000:2.0:1777502889.042183:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562109824:0@lo:38:kworker.0 00000100:00100000:0.0:1777502889.042196:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562109696 00000100:00100000:0.0:1777502889.042200:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562109696 00000100:00100000:2.0:1777502889.042213:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502889.042272:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562109824 00000100:00100000:0.0:1777502889.042280:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562109824 00000100:00100000:0.0:1777502889.042306:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16802 00000100:00100000:0.0:1777502889.042312:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc5ebf7800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562109696:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502889.042326:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502889.042330:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16803 00000100:00100000:2.0:1777502889.042337:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc5ebf5880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562109824:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502889.042348:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502889.042358:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc5ebf7800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562109696:12345-0@lo:38:kworker.0 Request processed in 46us (207us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502889.042364:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16802 00000100:00100000:2.0:1777502889.042390:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc5ebf5880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562109824:12345-0@lo:38:kworker.0 Request processed in 50us (176us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502889.042399:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16803 00000100:00080000:2.0:1777502889.042438:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502889.042443:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502889.042446:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562109824:0@lo:38:kworker.0 00000100:00080000:2.0:1777502889.042472:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502889.042476:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502889.042478:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562109696:0@lo:38:kworker.0 00010000:00010000:0.0:1777502889.425710:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502890.449823:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502891.473743:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502892.497819:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502893.521705:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502894.161760:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502894.161766:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502894.161771:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502894.161779:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9016 00000100:00080000:2.0:1777502894.161797:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502894.161805:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502894.161832:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc5ebf4700 x1863837562110080/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502894.161851:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502894.161855:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502894.161858:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502894.161863:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9016 00000100:00080000:2.0:1777502894.161866:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502894.161871:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502894.161887:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa35500 x1863837562110208/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502894.161919:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562109952:0@lo:400:kworker.0 00000100:00100000:2.0:1777502894.161989:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502894.162069:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562109952 00000100:00100000:0.0:1777502894.162078:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562109952 00000100:00100000:0.0:1777502894.162127:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 466 00000100:00100000:0.0:1777502894.162139:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1fc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562109952:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502894.162210:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1fc50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562109952:12345-0@lo:400:kworker.0 Request processed in 74us (220us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502894.162218:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 466 00000100:00100000:0.0:1777502894.162258:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562110080:0@lo:38:kworker.0 00000100:00100000:0.0:1777502894.162293:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502894.162321:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562110208:0@lo:38:kworker.0 00000100:00100000:3.0:1777502894.162329:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562109952:0@lo:400:kworker.0 00000100:00100000:0.0:1777502894.162347:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502894.162399:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562110080 00000100:00100000:0.0:1777502894.162403:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562110080 00000100:00100000:0.0:1777502894.162424:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562110208 00000100:00100000:0.0:1777502894.162427:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562110208 00000100:00100000:0.0:1777502894.162447:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16804 00000100:00100000:0.0:1777502894.162452:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4319e300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562110080:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502894.162463:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502894.162477:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16805 00000100:00100000:3.0:1777502894.162486:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4319c380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562110208:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502894.162497:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502894.162521:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4319e300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562110080:12345-0@lo:38:kworker.0 Request processed in 66us (226us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502894.162530:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16804 00000100:00100000:3.0:1777502894.162541:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4319c380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562110208:12345-0@lo:38:kworker.0 Request processed in 54us (195us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502894.162549:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16805 00000100:00080000:0.0:1777502894.162601:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502894.162606:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502894.162609:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562110080:0@lo:38:kworker.0 00000100:00080000:0.0:1777502894.162623:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502894.162627:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502894.162630:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562110208:0@lo:38:kworker.0 00010000:00010000:0.0:1777502894.545710:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502895.569743:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502896.593750:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502897.617675:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502898.641737:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502899.281756:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502899.281762:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502899.281766:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502899.281775:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9021 00000100:00080000:3.0:1777502899.281780:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502899.281787:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502899.281812:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76ed80 x1863837562110464/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502899.281831:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502899.281835:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502899.281838:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502899.281842:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9021 00000100:00080000:3.0:1777502899.281845:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502899.281850:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502899.281866:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76f480 x1863837562110592/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502899.281912:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562110336:0@lo:400:kworker.0 00000100:00100000:3.0:1777502899.282006:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502899.282090:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562110464:0@lo:38:kworker.0 00000100:00100000:3.0:1777502899.282136:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502899.282164:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562110592:0@lo:38:kworker.0 00000100:00100000:3.0:1777502899.282197:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502899.282238:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562110336 00000100:00100000:2.0:1777502899.282243:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562110464 00000100:00100000:3.0:1777502899.282243:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562110336 00000100:00100000:2.0:1777502899.282249:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562110464 00000100:00100000:3.0:1777502899.282290:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 467 00000100:00100000:3.0:1777502899.282299:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1d850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562110336:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502899.282323:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562110592 00000100:00100000:2.0:1777502899.282328:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562110592 00000100:00100000:3.0:1777502899.282354:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1d850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562110336:12345-0@lo:400:kworker.0 Request processed in 56us (350us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502899.282361:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 467 00000100:00100000:2.0:1777502899.282400:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16806 00000100:00100000:2.0:1777502899.282408:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562110464:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502899.282417:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562110336:0@lo:400:kworker.0 00010000:02000000:2.0:1777502899.282422:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502899.282467:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562110464:12345-0@lo:38:kworker.0 Request processed in 58us (330us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502899.282474:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16806 00000100:00100000:3.0:1777502899.282535:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16807 00000100:00100000:3.0:1777502899.282540:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76c700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562110592:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502899.282546:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502899.282575:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76c700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562110592:12345-0@lo:38:kworker.0 Request processed in 34us (378us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502899.282599:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16807 00000100:00080000:2.0:1777502899.282636:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502899.282648:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502899.282653:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562110464:0@lo:38:kworker.0 00000100:00080000:2.0:1777502899.282669:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502899.282672:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502899.282676:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562110592:0@lo:38:kworker.0 00010000:00010000:3.0:1777502899.665729:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502900.689743:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502901.713691:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502902.737741:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502903.761726:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502904.401751:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502904.401757:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502904.401761:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502904.401769:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9026 00000100:00080000:2.0:1777502904.401774:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502904.401780:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502904.401804:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa35500 x1863837562110848/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502904.401823:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502904.401826:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502904.401828:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502904.401833:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9026 00000100:00080000:2.0:1777502904.401835:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502904.401840:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502904.401855:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc560cb800 x1863837562110976/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502904.401883:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562110848:0@lo:38:kworker.0 00000100:00100000:2.0:1777502904.401944:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502904.402024:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc560cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562110976:0@lo:38:kworker.0 00000100:00100000:2.0:1777502904.402054:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502904.402123:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562110848 00000100:00100000:0.0:1777502904.402124:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562110720:0@lo:400:kworker.0 00000100:00100000:3.0:1777502904.402132:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562110848 00000100:00100000:0.0:1777502904.402172:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502904.402209:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562110976 00000100:00100000:2.0:1777502904.402214:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562110976 00000100:00100000:0.0:1777502904.402243:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16808 00000100:00100000:0.0:1777502904.402255:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc560c9c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562110848:12345-0@lo:38:kworker.0 00000100:00100000:2.0:1777502904.402255:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16809 00000100:00100000:2.0:1777502904.402265:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc560c8000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562110976:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502904.402275:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:2.0:1777502904.402277:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502904.402323:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc560c8000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562110976:12345-0@lo:38:kworker.0 Request processed in 58us (269us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502904.402333:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16809 00000100:00100000:0.0:1777502904.402393:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562110720 00000100:00100000:0.0:1777502904.402398:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562110720 00000100:00080000:3.0:1777502904.402423:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1777502904.402442:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 468 00000100:00100000:0.0:1777502904.402453:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1e050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562110720:12345-0@lo:400:kworker.0 00000100:00080000:3.0:1777502904.402469:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502904.402474:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc560cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562110976:0@lo:38:kworker.0 00000100:00100000:0.0:1777502904.402509:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1e050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562110720:12345-0@lo:400:kworker.0 Request processed in 59us (338us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502904.402515:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 468 00000100:00100000:0.0:1777502904.402572:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562110720:0@lo:400:kworker.0 00000100:00100000:0.0:1777502904.402663:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc560c9c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562110848:12345-0@lo:38:kworker.0 Request processed in 411us (720us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502904.402670:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16808 00000100:00080000:0.0:1777502904.402706:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502904.402710:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502904.402714:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562110848:0@lo:38:kworker.0 00010000:00010000:0.0:1777502904.785775:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502905.809697:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502906.833730:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502907.857698:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502908.881779:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502909.521758:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502909.521765:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502909.521769:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502909.521788:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9031 00000100:00080000:0.0:1777502909.521793:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502909.521801:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502909.521829:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa35500 x1863837562111232/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502909.521848:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502909.521852:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502909.521855:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502909.521861:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9031 00000100:00080000:0.0:1777502909.521864:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502909.521869:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502909.521886:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa35c00 x1863837562111360/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502909.521936:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562111104:0@lo:400:kworker.0 00000100:00100000:0.0:1777502909.522010:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502909.522119:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562111104 00000100:00100000:0.0:1777502909.522127:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562111104 00000100:00100000:0.0:1777502909.522180:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 469 00000100:00100000:0.0:1777502909.522191:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1d050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562111104:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502909.522252:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1d050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562111104:12345-0@lo:400:kworker.0 Request processed in 64us (242us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502909.522261:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 469 00000100:00100000:0.0:1777502909.522468:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562111104:0@lo:400:kworker.0 00000100:00100000:2.0:1777502909.522472:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562111232:0@lo:38:kworker.0 00000100:00100000:2.0:1777502909.522515:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502909.522545:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562111360:0@lo:38:kworker.0 00000100:00100000:3.0:1777502909.522553:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562111232 00000100:00100000:3.0:1777502909.522559:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562111232 00000100:00100000:2.0:1777502909.522575:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502909.522620:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562111360 00000100:00100000:3.0:1777502909.522625:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562111360 00000100:00100000:2.0:1777502909.522659:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16810 00000100:00100000:2.0:1777502909.522667:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc560cb800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562111232:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502909.522683:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502909.522729:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc560cb800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562111232:12345-0@lo:38:kworker.0 Request processed in 61us (213us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502909.522737:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16810 00000100:00100000:2.0:1777502909.522763:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16811 00000100:00100000:2.0:1777502909.522770:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f669f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562111360:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502909.522778:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502909.522809:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f669f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562111360:12345-0@lo:38:kworker.0 Request processed in 39us (235us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502909.522817:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16811 00000100:00080000:2.0:1777502909.522858:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502909.522865:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502909.522869:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562111232:0@lo:38:kworker.0 00000100:00080000:2.0:1777502909.522884:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502909.522889:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502909.522893:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562111360:0@lo:38:kworker.0 00010000:00010000:2.0:1777502909.905704:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502910.929741:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502911.953730:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502912.977770:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502914.001713:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502914.641754:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502914.641760:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502914.641765:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502914.641773:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9036 00000100:00080000:2.0:1777502914.641778:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502914.641784:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502914.641808:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f66bb80 x1863837562111616/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502914.641828:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502914.641831:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502914.641834:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502914.641839:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9036 00000100:00080000:2.0:1777502914.641841:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502914.641846:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502914.641861:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f66ad80 x1863837562111744/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502914.641892:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562111616:0@lo:38:kworker.0 00000100:00100000:2.0:1777502914.641994:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502914.642052:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562111744:0@lo:38:kworker.0 00000100:00100000:3.0:1777502914.642067:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562111488:0@lo:400:kworker.0 00000100:00100000:2.0:1777502914.642083:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502914.642112:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502914.642156:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562111616 00000100:00100000:2.0:1777502914.642164:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562111616 00000100:00100000:3.0:1777502914.642168:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562111744 00000100:00100000:3.0:1777502914.642173:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562111744 00000100:00100000:2.0:1777502914.642208:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16812 00000100:00100000:3.0:1777502914.642215:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16813 00000100:00100000:2.0:1777502914.642219:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f668700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562111616:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502914.642225:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f669500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562111744:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502914.642239:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:3.0:1777502914.642240:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502914.642299:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f668700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562111616:12345-0@lo:38:kworker.0 Request processed in 81us (306us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502914.642318:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16812 00000100:00100000:3.0:1777502914.642340:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f669500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562111744:12345-0@lo:38:kworker.0 Request processed in 75us (217us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502914.642347:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16813 00000100:00080000:2.0:1777502914.642366:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502914.642373:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502914.642379:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562111616:0@lo:38:kworker.0 00000100:00080000:2.0:1777502914.642398:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1777502914.642400:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562111488 00000100:00080000:2.0:1777502914.642404:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502914.642405:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562111488 00000100:00100000:2.0:1777502914.642410:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562111744:0@lo:38:kworker.0 00000100:00100000:3.0:1777502914.642444:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 470 00000100:00100000:3.0:1777502914.642452:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1e450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562111488:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502914.642493:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1e450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562111488:12345-0@lo:400:kworker.0 Request processed in 43us (381us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502914.642500:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 470 00000100:00100000:2.0:1777502914.642648:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562111488:0@lo:400:kworker.0 00010000:00010000:0.0:1777502915.025735:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502916.049721:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502917.073716:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502918.098070:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502919.121786:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502919.761731:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502919.761737:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502919.761742:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502919.761750:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9041 00000100:00080000:0.0:1777502919.761755:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502919.761762:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502919.761786:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4fa35500 x1863837562112000/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502919.761805:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502919.761823:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502919.761826:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502919.761831:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9041 00000100:00080000:0.0:1777502919.761834:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502919.761839:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502919.761857:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4319df80 x1863837562112128/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502919.761904:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562112000:0@lo:38:kworker.0 00000100:00100000:0.0:1777502919.761981:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502919.762037:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4319df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562112128:0@lo:38:kworker.0 00000100:00100000:0.0:1777502919.762065:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502919.762071:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4fa35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562111872:0@lo:400:kworker.0 00000100:00100000:3.0:1777502919.762120:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502919.762126:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562112000 00000100:00100000:0.0:1777502919.762134:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562112000 00000100:00100000:0.0:1777502919.762169:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562112128 00000100:00100000:0.0:1777502919.762172:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562112128 00000100:00100000:0.0:1777502919.762201:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16814 00000100:00100000:0.0:1777502919.762208:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c53480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562112000:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502919.762227:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502919.762238:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16815 00000100:00100000:3.0:1777502919.762246:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c53100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562112128:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502919.762257:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502919.762294:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562111872 00000100:00100000:0.0:1777502919.762298:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562111872 00000100:00100000:3.0:1777502919.762299:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c53100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562112128:12345-0@lo:38:kworker.0 Request processed in 53us (235us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502919.762307:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16815 00000100:00100000:0.0:1777502919.762344:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 471 00000100:00100000:0.0:1777502919.762384:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee1c450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562111872:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502919.762438:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee1c450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562111872:12345-0@lo:400:kworker.0 Request processed in 86us (319us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502919.762447:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 471 00000100:00100000:2.0:1777502919.762545:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562111872:0@lo:400:kworker.0 00000100:00100000:0.0:1777502919.762551:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c53480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562112000:12345-0@lo:38:kworker.0 Request processed in 340us (569us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502919.762565:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16814 00000100:00080000:0.0:1777502919.762642:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502919.762648:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502919.762653:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4fa35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562112000:0@lo:38:kworker.0 00000100:00080000:0.0:1777502919.762669:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502919.762675:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502919.762679:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4319df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562112128:0@lo:38:kworker.0 00010000:00010000:3.0:1777502920.145760:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502921.169733:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502922.193729:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502923.217746:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502924.241738:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502924.881748:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502924.881754:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502924.881758:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502924.881765:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9047 00000100:00080000:2.0:1777502924.881780:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502924.881786:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502924.881803:0:3310:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1777502924.881805:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f66ad80 x1863837562112384/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502924.881822:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1777502924.881826:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562112256:0@lo:400:kworker.0 00000100:00080000:2.0:1777502924.881826:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502924.881827:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502924.881835:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9047 00000100:00080000:2.0:1777502924.881837:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502924.881840:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502924.881854:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f66bb80 x1863837562112512/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502924.881918:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502924.882006:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562112384:0@lo:38:kworker.0 00000100:00100000:0.0:1777502924.882049:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502924.882076:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562112512:0@lo:38:kworker.0 00000100:00100000:0.0:1777502924.882111:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502924.882158:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562112256 00000100:00100000:3.0:1777502924.882166:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562112256 00000100:00100000:0.0:1777502924.882175:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562112384 00000100:00100000:0.0:1777502924.882180:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562112384 00000100:00100000:0.0:1777502924.882217:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562112512 00000100:00100000:3.0:1777502924.882217:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 472 00000100:00100000:0.0:1777502924.882222:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562112512 00000100:00100000:3.0:1777502924.882230:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee26050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562112256:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502924.882250:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16816 00000100:00100000:0.0:1777502924.882257:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd70c51500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562112384:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502924.882274:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502924.882296:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee26050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562112256:12345-0@lo:400:kworker.0 Request processed in 65us (376us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502924.882304:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 472 00000100:00100000:0.0:1777502924.882327:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd70c51500 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562112384:12345-0@lo:38:kworker.0 Request processed in 65us (276us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502924.882341:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16817 00000100:00100000:0.0:1777502924.882350:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16816 00000100:00100000:2.0:1777502924.882380:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4319df80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562112512:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502924.882391:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502924.882429:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4319df80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562112512:12345-0@lo:38:kworker.0 Request processed in 78us (318us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502924.882436:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16817 00000100:00080000:0.0:1777502924.882525:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502924.882533:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502924.882538:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562112384:0@lo:38:kworker.0 00000100:00100000:2.0:1777502924.882555:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562112256:0@lo:400:kworker.0 00000100:00080000:0.0:1777502924.882559:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502924.882564:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502924.882569:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562112512:0@lo:38:kworker.0 00010000:00010000:0.0:1777502925.265731:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502926.289684:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502927.314087:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502928.337716:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502929.361698:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502930.001749:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1777502930.001754:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562112640:0@lo:400:kworker.0 00000100:00080000:0.0:1777502930.001756:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502930.001762:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502930.001769:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9052 00000100:00080000:0.0:1777502930.001773:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502930.001780:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502930.001806:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd70c6b800 x1863837562112768/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502930.001835:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502930.001840:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502930.001842:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1777502930.001848:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:0.0:1777502930.001851:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9052 00000100:00080000:0.0:1777502930.001855:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502930.001860:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502930.001878:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd6e6ead80 x1863837562112896/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502930.001976:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562112768:0@lo:38:kworker.0 00000100:00100000:0.0:1777502930.002011:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502930.002033:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd6e6ead80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562112896:0@lo:38:kworker.0 00000100:00100000:0.0:1777502930.002065:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502930.002115:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562112640 00000100:00100000:0.0:1777502930.002121:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562112640 00000100:00100000:0.0:1777502930.002168:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 473 00000100:00100000:0.0:1777502930.002178:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee25050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562112640:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502930.002202:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562112768 00000100:00100000:2.0:1777502930.002208:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562112768 00000100:00100000:0.0:1777502930.002241:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee25050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562112640:12345-0@lo:400:kworker.0 Request processed in 64us (394us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502930.002252:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 473 00000100:00100000:2.0:1777502930.002323:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562112896 00000100:00100000:2.0:1777502930.002333:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562112896 00000100:00100000:2.0:1777502930.002399:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16818 00000100:00100000:2.0:1777502930.002404:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd6e6eb800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562112768:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502930.002418:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502930.002439:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562112640:0@lo:400:kworker.0 00000100:00100000:2.0:1777502930.002461:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd6e6eb800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562112768:12345-0@lo:38:kworker.0 Request processed in 54us (448us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502930.002467:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16818 00000100:00080000:2.0:1777502930.002536:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502930.002541:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502930.002545:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd70c6b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562112768:0@lo:38:kworker.0 00000100:00100000:3.0:1777502930.002569:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16819 00000100:00100000:3.0:1777502930.002592:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc492e4a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562112896:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502930.002601:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502930.002639:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc492e4a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562112896:12345-0@lo:38:kworker.0 Request processed in 63us (577us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502930.002646:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16819 00000100:00080000:2.0:1777502930.002694:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502930.002699:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502930.002703:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd6e6ead80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562112896:0@lo:38:kworker.0 00010000:00010000:2.0:1777502930.385718:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502931.409744:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502932.433746:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502933.457730:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502934.481684:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502935.121742:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502935.121748:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502935.121752:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502935.121774:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9057 00000100:00080000:3.0:1777502935.121790:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502935.121797:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502935.121822:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76ca80 x1863837562113152/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502935.121836:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00080000:3.0:1777502935.121842:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502935.121846:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502935.121848:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502935.121853:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9057 00000100:00100000:2.0:1777502935.121856:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562113024:0@lo:400:kworker.0 00000100:00080000:3.0:1777502935.121857:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502935.121864:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502935.121879:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76e300 x1863837562113280/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502935.121921:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502935.121949:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562113024 00000100:00100000:3.0:1777502935.121956:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562113024 00000100:00100000:2.0:1777502935.121981:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562113152:0@lo:38:kworker.0 00000100:00100000:3.0:1777502935.122007:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 474 00000100:00100000:3.0:1777502935.122017:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee26450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562113024:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502935.122024:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502935.122054:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562113280:0@lo:38:kworker.0 00000100:00100000:3.0:1777502935.122080:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee26450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562113024:12345-0@lo:400:kworker.0 Request processed in 65us (159us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502935.122091:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 474 00000100:00100000:2.0:1777502935.122095:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502935.122178:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562113152 00000100:00100000:2.0:1777502935.122182:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562113152 00000100:00100000:2.0:1777502935.122205:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562113280 00000100:00100000:2.0:1777502935.122207:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562113280 00000100:00100000:2.0:1777502935.122233:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16820 00000100:00100000:2.0:1777502935.122247:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f66a300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562113152:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502935.122263:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502935.122310:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f66a300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562113152:12345-0@lo:38:kworker.0 Request processed in 62us (289us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502935.122318:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16820 00000100:00100000:2.0:1777502935.122340:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16821 00000100:00100000:2.0:1777502935.122345:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f668a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562113280:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502935.122352:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502935.122382:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f668a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562113280:12345-0@lo:38:kworker.0 Request processed in 36us (294us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502935.122388:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16821 00000100:00100000:2.0:1777502935.122717:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562113024:0@lo:400:kworker.0 00000100:00080000:2.0:1777502935.122915:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502935.122922:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502935.122927:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562113152:0@lo:38:kworker.0 00000100:00080000:2.0:1777502935.122942:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502935.122947:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502935.122952:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562113280:0@lo:38:kworker.0 00010000:00010000:0.0:1777502935.505728:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502936.529733:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502937.553700:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502938.577718:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502939.601751:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502940.241773:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502940.241777:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502940.241781:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502940.241789:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9062 00000100:00080000:0.0:1777502940.241793:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502940.241799:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502940.241819:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc492e7b80 x1863837562113536/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502940.241885:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502940.241889:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502940.241891:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502940.241897:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9062 00000100:00080000:0.0:1777502940.241900:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502940.241905:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502940.241922:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc492e4e00 x1863837562113664/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502940.241944:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:0.0:1777502940.241959:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc492e4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562113408:0@lo:400:kworker.0 00000100:00100000:0.0:1777502940.242038:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502940.242108:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562113408 00000100:00100000:0.0:1777502940.242115:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562113408 00000100:00100000:2.0:1777502940.242133:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc492e7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562113536:0@lo:38:kworker.0 00000100:00100000:0.0:1777502940.242170:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 475 00000100:00100000:2.0:1777502940.242174:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502940.242186:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee24450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562113408:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502940.242199:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc492e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562113664:0@lo:38:kworker.0 00000100:00100000:2.0:1777502940.242234:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502940.242257:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee24450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562113408:12345-0@lo:400:kworker.0 Request processed in 74us (218us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502940.242266:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 475 00000100:00100000:0.0:1777502940.242337:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc492e4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562113408:0@lo:400:kworker.0 00000100:00100000:2.0:1777502940.242339:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562113536 00000100:00100000:2.0:1777502940.242345:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562113536 00000100:00100000:2.0:1777502940.242378:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562113664 00000100:00100000:2.0:1777502940.242381:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562113664 00000100:00100000:2.0:1777502940.242403:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16822 00000100:00100000:2.0:1777502940.242410:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f669c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562113536:12345-0@lo:38:kworker.0 00000100:00100000:0.0:1777502940.242413:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16823 00000100:00100000:0.0:1777502940.242422:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f668000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562113664:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502940.242427:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:0.0:1777502940.242434:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502940.242464:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f669c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562113536:12345-0@lo:38:kworker.0 Request processed in 53us (292us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502940.242471:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16822 00000100:00100000:0.0:1777502940.242482:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f668000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562113664:12345-0@lo:38:kworker.0 Request processed in 58us (246us total) trans 0 rc -19/-19 00000100:00080000:2.0:1777502940.242502:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502940.242506:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502940.242511:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc492e7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562113536:0@lo:38:kworker.0 00000100:00100000:0.0:1777502940.242524:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16823 00000100:00080000:2.0:1777502940.242525:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502940.242531:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502940.242534:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc492e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562113664:0@lo:38:kworker.0 00010000:00010000:3.0:1777502940.625757:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502941.649722:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502942.673683:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502943.697793:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502944.721757:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502945.361727:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502945.361735:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502945.361739:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502945.361747:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9067 00000100:00080000:0.0:1777502945.361751:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502945.361758:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502945.361779:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc492e5180 x1863837562113920/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502945.361796:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502945.361799:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502945.361801:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502945.361805:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9067 00000100:00080000:0.0:1777502945.361807:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502945.361810:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502945.361822:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc492e4e00 x1863837562114048/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502945.361852:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc492e4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562113792:0@lo:400:kworker.0 00000100:00100000:0.0:1777502945.361923:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502945.362006:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc492e5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562113920:0@lo:38:kworker.0 00000100:00100000:2.0:1777502945.362046:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502945.362077:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562113792 00000100:00100000:3.0:1777502945.362086:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562113792 00000100:00100000:2.0:1777502945.362088:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc492e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562114048:0@lo:38:kworker.0 00000100:00100000:2.0:1777502945.362115:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502945.362137:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 476 00000100:00100000:2.0:1777502945.362160:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562113920 00000100:00100000:2.0:1777502945.362163:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562113920 00000100:00100000:2.0:1777502945.362193:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562114048 00000100:00100000:2.0:1777502945.362195:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562114048 00000100:00100000:3.0:1777502945.362202:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee24850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562113792:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502945.362219:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16824 00000100:00100000:2.0:1777502945.362226:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f669180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562113920:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502945.362241:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502945.362285:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f669180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562113920:12345-0@lo:38:kworker.0 Request processed in 58us (239us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502945.362286:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee24850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562113792:12345-0@lo:400:kworker.0 Request processed in 88us (366us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502945.362297:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16825 00000100:00100000:3.0:1777502945.362297:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 476 00000100:00100000:2.0:1777502945.362309:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16824 00000100:00100000:0.0:1777502945.362310:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f668e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562114048:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502945.362321:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00080000:2.0:1777502945.362369:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1777502945.362375:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f668e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562114048:12345-0@lo:38:kworker.0 Request processed in 67us (260us total) trans 0 rc -19/-19 00000100:00080000:2.0:1777502945.362376:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502945.362383:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc492e5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562113920:0@lo:38:kworker.0 00000100:00100000:3.0:1777502945.362383:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc492e4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562113792:0@lo:400:kworker.0 00000100:00100000:0.0:1777502945.362388:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16825 00000100:00080000:2.0:1777502945.362402:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502945.362407:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502945.362410:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc492e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562114048:0@lo:38:kworker.0 00010000:00010000:0.0:1777502945.745763:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502946.769675:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502947.793741:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502948.817709:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502949.841744:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502950.481866:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502950.481872:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502950.481876:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502950.481884:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9072 00000100:00080000:3.0:1777502950.481890:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502950.481897:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502950.481920:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76ca80 x1863837562114304/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502950.481940:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502950.481943:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502950.481945:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502950.481949:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9072 00000100:00080000:3.0:1777502950.481952:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502950.481956:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502950.481969:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76d880 x1863837562114432/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502950.481992:0:3309:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1777502950.482009:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562114176:0@lo:400:kworker.0 00000100:00100000:2.0:1777502950.482192:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562114304:0@lo:38:kworker.0 00000100:00100000:2.0:1777502950.482246:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502950.482288:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562114432:0@lo:38:kworker.0 00000100:00100000:2.0:1777502950.482316:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502950.482358:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562114304 00000100:00100000:2.0:1777502950.482365:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562114304 00000100:00100000:0.0:1777502950.482382:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562114432 00000100:00100000:0.0:1777502950.482389:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562114432 00000100:00100000:2.0:1777502950.482400:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16826 00000100:00100000:2.0:1777502950.482407:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f66b480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562114304:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502950.482421:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502950.482459:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f66b480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562114304:12345-0@lo:38:kworker.0 Request processed in 52us (213us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502950.482465:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16826 00000100:00100000:2.0:1777502950.482491:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16827 00000100:00100000:2.0:1777502950.482496:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc4fb6b050 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562114432:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502950.482503:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502950.482536:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc4fb6b050 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562114432:12345-0@lo:38:kworker.0 Request processed in 38us (218us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502950.482553:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16827 00000100:00080000:0.0:1777502950.483065:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502950.483090:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502950.483095:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562114304:0@lo:38:kworker.0 00000100:00080000:0.0:1777502950.483112:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502950.483117:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502950.483122:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562114432:0@lo:38:kworker.0 00000100:00100000:3.0:1777502950.483807:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502950.483847:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562114176 00000100:00100000:0.0:1777502950.483853:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562114176 00000100:00100000:0.0:1777502950.483895:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 477 00000100:00100000:0.0:1777502950.483904:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee27050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562114176:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502950.483956:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee27050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562114176:12345-0@lo:400:kworker.0 Request processed in 53us (148us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502950.483963:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 477 00000100:00100000:3.0:1777502950.483986:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562114176:0@lo:400:kworker.0 00010000:00010000:2.0:1777502950.865733:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502951.889807:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502952.913804:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777502953.937731:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502954.961724:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502955.601928:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502955.601933:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502955.601937:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502955.601944:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9077 00000100:00080000:3.0:1777502955.601949:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502955.601955:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502955.601974:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76f480 x1863837562114688/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502955.601986:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502955.601988:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502955.601990:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502955.601993:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9077 00000100:00080000:3.0:1777502955.601996:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502955.601999:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502955.602009:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76ed80 x1863837562114816/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502955.602042:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:3.0:1777502955.602060:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562114560:0@lo:400:kworker.0 00000100:00100000:3.0:1777502955.602136:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502955.602217:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562114560 00000100:00100000:3.0:1777502955.602224:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562114560 00000100:00100000:3.0:1777502955.602278:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 478 00000100:00100000:3.0:1777502955.602303:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee25c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562114560:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502955.602350:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee25c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562114560:12345-0@lo:400:kworker.0 Request processed in 51us (216us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502955.602356:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 478 00000100:00100000:3.0:1777502955.602421:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562114560:0@lo:400:kworker.0 00000100:00100000:0.0:1777502955.602818:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562114688:0@lo:38:kworker.0 00000100:00100000:0.0:1777502955.602849:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502955.602874:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562114816:0@lo:38:kworker.0 00000100:00100000:0.0:1777502955.602893:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502955.603044:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562114688 00000100:00100000:3.0:1777502955.603050:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562114688 00000100:00100000:2.0:1777502955.603068:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562114816 00000100:00100000:2.0:1777502955.603074:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562114816 00000100:00100000:3.0:1777502955.603081:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16828 00000100:00100000:3.0:1777502955.603087:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc492e7b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562114688:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502955.603102:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502955.603110:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16829 00000100:00100000:2.0:1777502955.603117:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc492e4a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562114816:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502955.603128:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502955.603145:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc492e7b80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562114688:12345-0@lo:38:kworker.0 Request processed in 56us (294us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502955.603153:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16828 00000100:00100000:2.0:1777502955.603171:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc492e4a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562114816:12345-0@lo:38:kworker.0 Request processed in 52us (276us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502955.603178:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16829 00000100:00080000:0.0:1777502955.603210:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502955.603217:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502955.603222:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562114688:0@lo:38:kworker.0 00000100:00080000:0.0:1777502955.603238:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502955.603241:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502955.603245:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562114816:0@lo:38:kworker.0 00010000:00010000:0.0:1777502955.985761:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502957.009793:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502958.033780:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502959.057742:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502960.081798:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502960.721838:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502960.721924:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502960.721928:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502960.721937:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9082 00000100:00080000:2.0:1777502960.721942:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502960.721949:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502960.722004:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f668700 x1863837562115072/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502960.722030:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502960.722033:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502960.722035:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502960.722048:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9082 00000100:00080000:2.0:1777502960.722051:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502960.722055:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502960.722091:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f669500 x1863837562115200/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502960.722117:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:2.0:1777502960.722156:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f669f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562114944:0@lo:400:kworker.0 00000100:00100000:2.0:1777502960.722234:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502960.722307:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f668700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562115072:0@lo:38:kworker.0 00000100:00100000:1.0F:1777502960.722332:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562114944 00000100:00100000:1.0:1777502960.722359:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562114944 00000100:00100000:2.0:1777502960.722365:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502960.722407:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f669500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562115200:0@lo:38:kworker.0 00000100:00100000:1.0:1777502960.722424:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 479 00000100:00100000:1.0:1777502960.722436:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee25450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562114944:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502960.722441:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502960.722474:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562115072 00000100:00100000:2.0:1777502960.722478:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562115072 00000100:00100000:2.0:1777502960.722513:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562115200 00000100:00100000:2.0:1777502960.722516:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562115200 00000100:00100000:1.0:1777502960.722521:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee25450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562114944:12345-0@lo:400:kworker.0 Request processed in 88us (287us total) trans 0 rc 0/0 00000100:00100000:1.0:1777502960.722529:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 479 00000100:00100000:2.0:1777502960.722561:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16830 00000100:00100000:2.0:1777502960.722567:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f66a300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562115072:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502960.722616:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:1.0:1777502960.722618:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f669f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562114944:0@lo:400:kworker.0 00000100:00100000:2.0:1777502960.722655:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f66a300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562115072:12345-0@lo:38:kworker.0 Request processed in 87us (290us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502960.722662:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16830 00000100:00100000:1.0:1777502960.723994:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16831 00000100:00100000:1.0:1777502960.724000:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f668a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562115200:12345-0@lo:38:kworker.0 00010000:02000000:1.0:1777502960.724009:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:1.0:1777502960.724040:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f668a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562115200:12345-0@lo:38:kworker.0 Request processed in 39us (1602us total) trans 0 rc -19/-19 00000100:00080000:2.0:1777502960.724044:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1777502960.724045:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16831 00000100:00080000:2.0:1777502960.724053:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502960.724058:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f668700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562115072:0@lo:38:kworker.0 00000100:00080000:2.0:1777502960.724068:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502960.724071:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502960.724073:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f669500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562115200:0@lo:38:kworker.0 00010000:00010000:3.0:1777502961.106289:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502962.129740:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502963.153783:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502964.178066:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502965.201797:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502965.841737:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502965.841743:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502965.841748:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502965.841756:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9087 00000100:00080000:2.0:1777502965.841761:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502965.841769:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502965.841794:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f668700 x1863837562115456/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502965.841814:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502965.841818:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502965.841828:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502965.841833:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9087 00000100:00080000:2.0:1777502965.841836:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502965.841841:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502965.841857:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f669c00 x1863837562115584/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502965.841883:0:3310:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1777502965.841904:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f669500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562115328:0@lo:400:kworker.0 00000100:00100000:2.0:1777502965.842018:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502965.842104:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f668700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562115456:0@lo:38:kworker.0 00000100:00100000:2.0:1777502965.842120:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562115328 00000100:00100000:2.0:1777502965.842128:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562115328 00000100:00100000:3.0:1777502965.842154:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502965.842182:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 480 00000100:00100000:2.0:1777502965.842195:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee27450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562115328:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502965.842201:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f669c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562115584:0@lo:38:kworker.0 00000100:00100000:3.0:1777502965.842233:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502965.842270:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee27450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562115328:12345-0@lo:400:kworker.0 Request processed in 78us (251us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502965.842297:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 480 00000100:00100000:3.0:1777502965.842301:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562115456 00000100:00100000:3.0:1777502965.842307:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562115456 00000100:00100000:3.0:1777502965.842347:0:186937:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562115584 00000100:00100000:3.0:1777502965.842357:0:186937:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562115584 00000100:00100000:2.0:1777502965.842376:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16832 00000100:00100000:2.0:1777502965.842384:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76e300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562115456:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502965.842390:0:186937:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16833 00000100:00100000:3.0:1777502965.842399:0:186937:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76ed80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562115584:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502965.842402:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:3.0:1777502965.842412:0:186937:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502965.842468:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76e300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562115456:12345-0@lo:38:kworker.0 Request processed in 82us (314us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502965.842470:0:186937:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76ed80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_001:0+-99:3308:x1863837562115584:12345-0@lo:38:kworker.0 Request processed in 71us (237us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502965.842496:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16832 00000100:00100000:3.0:1777502965.842503:0:186937:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16833 00000100:00080000:3.0:1777502965.842569:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502965.842575:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502965.842606:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f668700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562115456:0@lo:38:kworker.0 00000100:00080000:3.0:1777502965.842622:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502965.842627:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502965.842631:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f669c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562115584:0@lo:38:kworker.0 00000100:00100000:3.0:1777502965.842685:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f669500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562115328:0@lo:400:kworker.0 00010000:00010000:3.0:1777502966.225744:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502967.249713:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502968.274433:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502969.297898:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502970.321731:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502970.961797:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502970.961815:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502970.961819:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502970.961828:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9093 00000100:00080000:0.0:1777502970.961832:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502970.961849:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502970.961870:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc492e5180 x1863837562115840/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502970.961886:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502970.961889:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502970.961891:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502970.961895:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9093 00000100:00080000:0.0:1777502970.961897:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502970.961900:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502970.961910:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc492e4700 x1863837562115968/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502970.961928:0:3309:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1777502970.961942:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc492e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562115712:0@lo:400:kworker.0 00000100:00100000:0.0:1777502970.962058:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502970.962136:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc492e5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562115840:0@lo:38:kworker.0 00000100:00100000:0.0:1777502970.962183:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502970.962206:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc492e4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562115968:0@lo:38:kworker.0 00000100:00100000:0.0:1777502970.962226:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502970.962271:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562115712 00000100:00100000:0.0:1777502970.962276:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562115712 00000100:00100000:0.0:1777502970.962332:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 481 00000100:00100000:0.0:1777502970.962341:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee26850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562115712:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502970.962386:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee26850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562115712:12345-0@lo:400:kworker.0 Request processed in 48us (330us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502970.962394:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 481 00000100:00100000:0.0:1777502970.962481:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562115840 00000100:00100000:0.0:1777502970.962484:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562115840 00000100:00100000:0.0:1777502970.962513:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562115968 00000100:00100000:0.0:1777502970.962517:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562115968 00000100:00100000:3.0:1777502970.962546:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16834 00000100:00100000:0.0:1777502970.962557:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16835 00000100:00100000:3.0:1777502970.962558:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493f9180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562115840:12345-0@lo:38:kworker.0 00000100:00100000:0.0:1777502970.962567:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493f9c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562115968:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502970.962591:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:3.0:1777502970.962599:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502970.962626:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493f9c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562115968:12345-0@lo:38:kworker.0 Request processed in 60us (399us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502970.962632:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16835 00000100:00100000:3.0:1777502970.962656:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493f9180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562115840:12345-0@lo:38:kworker.0 Request processed in 100us (472us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502970.962665:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16834 00000100:00080000:0.0:1777502970.962674:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502970.962679:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502970.962682:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc492e5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562115840:0@lo:38:kworker.0 00000100:00080000:0.0:1777502970.962693:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502970.962717:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502970.962721:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc492e4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562115968:0@lo:38:kworker.0 00000100:00100000:3.0:1777502970.962751:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc492e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562115712:0@lo:400:kworker.0 00010000:00010000:2.0:1777502971.345695:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502972.369804:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502973.393712:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502974.417734:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502975.441706:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502976.081723:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502976.081730:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502976.081735:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1777502976.081737:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562116096:0@lo:400:kworker.0 00000100:00080000:3.0:1777502976.081742:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9098 00000100:00080000:3.0:1777502976.081747:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502976.081754:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502976.081776:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76d880 x1863837562116224/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502976.081797:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502976.081800:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502976.081802:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502976.081806:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9098 00000100:00080000:3.0:1777502976.081809:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502976.081812:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777502976.081824:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76ca80 x1863837562116352/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1777502976.081857:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502976.081859:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562116224:0@lo:38:kworker.0 00000100:00100000:3.0:1777502976.081908:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502976.081949:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562116352:0@lo:38:kworker.0 00000100:00100000:3.0:1777502976.081974:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502976.082025:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562116096 00000100:00100000:3.0:1777502976.082030:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562116096 00000100:00100000:2.0:1777502976.082070:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562116224 00000100:00100000:3.0:1777502976.082074:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 482 00000100:00100000:2.0:1777502976.082077:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562116224 00000100:00100000:3.0:1777502976.082083:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee27850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562116096:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502976.082120:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562116352 00000100:00100000:2.0:1777502976.082129:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562116352 00000100:00100000:3.0:1777502976.082148:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee27850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562116096:12345-0@lo:400:kworker.0 Request processed in 66us (293us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502976.082155:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 482 00000100:00100000:2.0:1777502976.082159:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16836 00000100:00100000:2.0:1777502976.082166:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76df80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562116224:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777502976.082182:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502976.082212:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16837 00000100:00100000:3.0:1777502976.082217:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76c380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562116352:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502976.082225:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502976.082234:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76df80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562116224:12345-0@lo:38:kworker.0 Request processed in 66us (324us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502976.082243:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16836 00000100:00100000:3.0:1777502976.082259:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76c380 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562116352:12345-0@lo:38:kworker.0 Request processed in 40us (285us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502976.082265:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16837 00000100:00080000:2.0:1777502976.082312:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502976.082320:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502976.082325:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562116224:0@lo:38:kworker.0 00000100:00080000:2.0:1777502976.082343:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777502976.082347:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777502976.082352:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562116352:0@lo:38:kworker.0 00000100:00100000:3.0:1777502976.082553:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562116096:0@lo:400:kworker.0 00010000:00010000:3.0:1777502976.465753:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502977.489717:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502978.513792:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502979.537700:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502980.561763:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777502981.201769:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502981.201778:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502981.201783:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502981.201794:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9103 00000100:00080000:0.0:1777502981.201800:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1777502981.201803:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00080000:0.0:1777502981.201810:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502981.201836:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493fa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562116480:0@lo:400:kworker.0 00000100:00100000:0.0:1777502981.201840:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc493fb100 x1863837562116608/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777502981.201868:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777502981.201872:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777502981.201875:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777502981.201882:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9103 00000100:00080000:0.0:1777502981.201887:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777502981.201893:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502981.201912:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc493fbb80 x1863837562116736/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777502981.201972:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493fb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562116608:0@lo:38:kworker.0 00000100:00100000:2.0:1777502981.202042:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502981.202051:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502981.202153:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493fbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562116736:0@lo:38:kworker.0 00000100:00100000:0.0:1777502981.202205:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502981.202370:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562116608 00000100:00100000:0.0:1777502981.202380:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562116608 00000100:00100000:0.0:1777502981.202565:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562116736 00000100:00100000:0.0:1777502981.202572:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562116736 00000100:00100000:0.0:1777502981.202640:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16838 00000100:00100000:2.0:1777502981.202649:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16839 00000100:00100000:0.0:1777502981.202650:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493fad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562116608:12345-0@lo:38:kworker.0 00000100:00100000:2.0:1777502981.202660:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493faa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562116736:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777502981.202670:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:2.0:1777502981.202672:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502981.202777:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493fad80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562116608:12345-0@lo:38:kworker.0 Request processed in 129us (745us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502981.202783:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493faa00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562116736:12345-0@lo:38:kworker.0 Request processed in 124us (579us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777502981.202788:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16838 00000100:00100000:2.0:1777502981.202794:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16839 00000100:00100000:0.0:1777502981.202841:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562116480 00000100:00100000:0.0:1777502981.202847:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562116480 00000100:00100000:0.0:1777502981.202905:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 483 00000100:00100000:0.0:1777502981.202923:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee24050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562116480:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777502981.202999:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee24050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562116480:12345-0@lo:400:kworker.0 Request processed in 82us (969us total) trans 0 rc 0/0 00000100:00100000:0.0:1777502981.203010:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 483 00000100:00080000:0.0:1777502981.203084:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502981.203092:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502981.203097:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493fb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562116608:0@lo:38:kworker.0 00000100:00080000:0.0:1777502981.203114:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502981.203120:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502981.203130:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493fbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562116736:0@lo:38:kworker.0 00000100:00100000:0.0:1777502981.203200:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493fa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562116480:0@lo:400:kworker.0 00010000:00010000:3.0:1777502981.585703:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502982.609753:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502983.633694:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502984.657719:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502985.681694:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502986.321860:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502986.321866:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502986.321870:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1777502986.321874:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f668000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562116864:0@lo:400:kworker.0 00000100:00080000:2.0:1777502986.321879:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9108 00000100:00080000:2.0:1777502986.321885:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502986.321891:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502986.321914:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f669180 x1863837562116992/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502986.321931:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502986.321934:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502986.321937:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502986.321941:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9108 00000100:00080000:2.0:1777502986.321946:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1777502986.321947:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00080000:2.0:1777502986.321952:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502986.321980:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f668e00 x1863837562117120/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502986.322027:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562116864 00000100:00100000:3.0:1777502986.322034:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562116864 00000100:00100000:3.0:1777502986.322082:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 484 00000100:00100000:3.0:1777502986.322102:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee26c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562116864:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502986.322166:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee26c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562116864:12345-0@lo:400:kworker.0 Request processed in 67us (214us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502986.322175:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 484 00000100:00100000:3.0:1777502986.322307:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f668000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562116864:0@lo:400:kworker.0 00000100:00100000:0.0:1777502986.322552:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f669180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562116992:0@lo:38:kworker.0 00000100:00100000:0.0:1777502986.322681:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777502986.322708:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f668e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562117120:0@lo:38:kworker.0 00000100:00100000:0.0:1777502986.322765:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502986.322903:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562116992 00000100:00100000:3.0:1777502986.322908:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562116992 00000100:00100000:3.0:1777502986.322932:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562117120 00000100:00100000:3.0:1777502986.322934:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562117120 00000100:00100000:3.0:1777502986.322953:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16840 00000100:00100000:3.0:1777502986.322958:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493fa300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562116992:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502986.322972:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502986.323009:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493fa300 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562116992:12345-0@lo:38:kworker.0 Request processed in 49us (341us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502986.323017:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16840 00000100:00100000:3.0:1777502986.323036:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16841 00000100:00100000:3.0:1777502986.323042:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493fbb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562117120:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502986.323050:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502986.323083:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493fbb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562117120:12345-0@lo:38:kworker.0 Request processed in 40us (318us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502986.323089:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16841 00000100:00080000:0.0:1777502986.323105:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502986.323113:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502986.323119:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f669180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562116992:0@lo:38:kworker.0 00000100:00080000:0.0:1777502986.323132:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502986.323136:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502986.323140:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f668e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562117120:0@lo:38:kworker.0 00010000:00010000:3.0:1777502986.705736:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502987.729705:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502988.753852:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502989.777795:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502990.801715:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777502991.441778:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502991.441784:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502991.441788:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502991.441797:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9113 00000100:00080000:2.0:1777502991.441802:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502991.441808:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502991.441832:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f66b800 x1863837562117376/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777502991.441852:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777502991.441856:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777502991.441858:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777502991.441863:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9113 00000100:00080000:2.0:1777502991.441866:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777502991.441870:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777502991.441885:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f668e00 x1863837562117504/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777502991.441915:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562117248:0@lo:400:kworker.0 00000100:00100000:2.0:1777502991.441995:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502991.442081:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562117248 00000100:00100000:2.0:1777502991.442088:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562117248 00000100:00100000:2.0:1777502991.442139:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 485 00000100:00100000:2.0:1777502991.442150:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee27c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562117248:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777502991.442223:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee27c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562117248:12345-0@lo:400:kworker.0 Request processed in 76us (229us total) trans 0 rc 0/0 00000100:00100000:2.0:1777502991.442232:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 485 00000100:00100000:3.0:1777502991.442298:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562117376:0@lo:38:kworker.0 00000100:00100000:2.0:1777502991.442307:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562117248:0@lo:400:kworker.0 00000100:00100000:3.0:1777502991.442338:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502991.442373:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f668e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562117504:0@lo:38:kworker.0 00000100:00100000:3.0:1777502991.442398:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777502991.442421:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562117376 00000100:00100000:2.0:1777502991.442426:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562117376 00000100:00100000:0.0:1777502991.442452:0:205372:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562117504 00000100:00100000:0.0:1777502991.442468:0:205372:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562117504 00000100:00100000:2.0:1777502991.442468:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16842 00000100:00100000:2.0:1777502991.442478:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76f480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562117376:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777502991.442490:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16843 00010000:02000000:2.0:1777502991.442495:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502991.442500:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76fb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562117504:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502991.442534:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777502991.442550:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76f480 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562117376:12345-0@lo:38:kworker.0 Request processed in 71us (211us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777502991.442558:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16842 00000100:00100000:3.0:1777502991.442574:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76fb80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562117504:12345-0@lo:38:kworker.0 Request processed in 74us (175us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502991.442602:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16843 00000100:00080000:0.0:1777502991.442622:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502991.442630:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502991.442635:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562117376:0@lo:38:kworker.0 00000100:00080000:0.0:1777502991.442649:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777502991.442654:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777502991.442658:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f668e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562117504:0@lo:38:kworker.0 00010000:00010000:2.0:1777502991.825719:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502992.849726:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502993.873798:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502994.897761:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777502995.921707:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777502996.561726:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502996.561731:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502996.561735:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502996.561741:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9118 00000100:00080000:3.0:1777502996.561744:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502996.561750:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502996.561753:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562117632:0@lo:400:kworker.0 00000100:00100000:3.0:1777502996.561768:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76e680 x1863837562117760/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777502996.561783:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777502996.561787:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777502996.561789:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777502996.561795:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9118 00000100:00080000:3.0:1777502996.561797:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777502996.561802:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777502996.561813:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502996.561819:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76c000 x1863837562117888/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777502996.561855:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562117760:0@lo:38:kworker.0 00000100:00100000:0.0:1777502996.561881:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562117632 00000100:00100000:0.0:1777502996.561887:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562117632 00000100:00100000:3.0:1777502996.561890:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502996.561934:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562117888:0@lo:38:kworker.0 00000100:00100000:0.0:1777502996.561940:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 486 00000100:00100000:0.0:1777502996.561954:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee25850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562117632:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777502996.561962:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777502996.562006:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562117760 00000100:00100000:0.0:1777502996.562011:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee25850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562117632:12345-0@lo:400:kworker.0 Request processed in 62us (198us total) trans 0 rc 0/0 00000100:00100000:3.0:1777502996.562011:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562117760 00000100:00100000:0.0:1777502996.562020:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 486 00000100:00100000:3.0:1777502996.562042:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562117888 00000100:00100000:3.0:1777502996.562046:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562117888 00000100:00100000:3.0:1777502996.562068:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16844 00000100:00100000:3.0:1777502996.562073:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562117760:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502996.562087:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777502996.562114:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562117632:0@lo:400:kworker.0 00000100:00100000:3.0:1777502996.562125:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76ea00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562117760:12345-0@lo:38:kworker.0 Request processed in 49us (235us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502996.562131:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16844 00000100:00100000:3.0:1777502996.562147:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16845 00000100:00100000:3.0:1777502996.562151:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76c700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562117888:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777502996.562156:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777502996.562182:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76c700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562117888:12345-0@lo:38:kworker.0 Request processed in 30us (221us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777502996.562187:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16845 00000100:00080000:3.0:1777502996.562257:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502996.562264:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502996.562269:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562117760:0@lo:38:kworker.0 00000100:00080000:3.0:1777502996.562289:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777502996.562295:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777502996.562300:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562117888:0@lo:38:kworker.0 00010000:00010000:3.0:1777502996.945729:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502997.969826:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777502998.993745:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777503000.017812:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777503001.041745:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777503001.682522:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777503001.682527:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777503001.682531:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777503001.682537:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9123 00000100:00080000:2.0:1777503001.682540:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777503001.682545:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777503001.682565:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f669180 x1863837562118144/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777503001.682576:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777503001.682592:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777503001.683649:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777503001.683653:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9123 00000100:00080000:2.0:1777503001.683657:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777503001.683661:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777503001.683678:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f668000 x1863837562118272/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777503001.683706:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:2.0:1777503001.683724:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562118016:0@lo:400:kworker.0 00000100:00100000:2.0:1777503001.683787:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777503001.683853:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562118016 00000100:00100000:2.0:1777503001.683858:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562118016 00000100:00100000:2.0:1777503001.683897:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 487 00000100:00100000:2.0:1777503001.683904:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7ee24c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562118016:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777503001.683946:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7ee24c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562118016:12345-0@lo:400:kworker.0 Request processed in 42us (159us total) trans 0 rc 0/0 00000100:00100000:2.0:1777503001.683953:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 487 00000100:00100000:2.0:1777503001.684012:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562118016:0@lo:400:kworker.0 00000100:00100000:3.0:1777503001.684132:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f669180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562118144:0@lo:38:kworker.0 00000100:00100000:3.0:1777503001.684177:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503001.684205:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f668000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562118272:0@lo:38:kworker.0 00000100:00100000:3.0:1777503001.684228:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503001.684262:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562118144 00000100:00100000:3.0:1777503001.684265:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562118144 00000100:00100000:3.0:1777503001.684283:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562118272 00000100:00100000:3.0:1777503001.684285:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562118272 00000100:00100000:3.0:1777503001.684307:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16846 00000100:00100000:3.0:1777503001.684313:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76c000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562118144:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777503001.684328:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777503001.684367:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76c000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562118144:12345-0@lo:38:kworker.0 Request processed in 52us (189us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777503001.684374:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16846 00000100:00100000:3.0:1777503001.684390:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16847 00000100:00100000:3.0:1777503001.684394:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f76e680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562118272:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777503001.684400:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777503001.684424:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f76e680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562118272:12345-0@lo:38:kworker.0 Request processed in 29us (196us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777503001.684428:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16847 00000100:00080000:3.0:1777503001.684474:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777503001.684480:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777503001.684485:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f669180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562118144:0@lo:38:kworker.0 00000100:00080000:3.0:1777503001.684500:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777503001.684505:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777503001.684509:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f668000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562118272:0@lo:38:kworker.0 00010000:00010000:3.0:1777503002.065750:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503003.089733:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777503004.113704:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777503005.137724:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503006.161742:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777503006.801732:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777503006.801739:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777503006.801744:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777503006.801752:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9128 00000100:00080000:3.0:1777503006.801757:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1777503006.801759:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562118400:0@lo:400:kworker.0 00000100:00080000:3.0:1777503006.801766:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777503006.801811:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76ca80 x1863837562118528/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777503006.801834:0:194113:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777503006.801838:0:194113:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777503006.801841:0:194113:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777503006.801863:0:194113:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9128 00000100:00080000:3.0:1777503006.801867:0:194113:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777503006.801872:0:194113:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777503006.801879:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503006.801892:0:194113:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd7f76d880 x1863837562118656/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777503006.801927:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562118528:0@lo:38:kworker.0 00000100:00100000:3.0:1777503006.801998:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503006.802056:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd7f76d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562118656:0@lo:38:kworker.0 00000100:00100000:3.0:1777503006.802119:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503006.802210:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562118528 00000100:00100000:3.0:1777503006.802217:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562118528 00000100:00100000:0.0:1777503006.802264:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562118400 00000100:00100000:0.0:1777503006.802270:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562118400 00000100:00100000:3.0:1777503006.802297:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16848 00000100:00100000:3.0:1777503006.802305:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc5ebf5c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562118528:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777503006.802323:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777503006.802379:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc5ebf5c00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562118528:12345-0@lo:38:kworker.0 Request processed in 73us (380us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777503006.802387:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16848 00000100:00100000:0.0:1777503006.802391:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 488 00000100:00100000:0.0:1777503006.802415:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f69b850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562118400:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777503006.802498:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562118656 00000100:00100000:3.0:1777503006.802502:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562118656 00000100:00100000:3.0:1777503006.802529:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16849 00000100:00100000:3.0:1777503006.802535:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc5ebf4000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562118656:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777503006.802545:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777503006.802598:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc5ebf4000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562118656:12345-0@lo:38:kworker.0 Request processed in 43us (460us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777503006.802606:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16849 00000100:00100000:0.0:1777503006.802757:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f69b850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562118400:12345-0@lo:400:kworker.0 Request processed in 357us (882us total) trans 0 rc 0/0 00000100:00080000:3.0:1777503006.802791:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777503006.802798:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777503006.802803:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562118528:0@lo:38:kworker.0 00000100:00080000:3.0:1777503006.802860:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777503006.802866:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777503006.802870:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562118656:0@lo:38:kworker.0 00000100:00100000:0.0:1777503006.803363:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 488 00000100:00100000:2.0:1777503006.803405:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd7f76f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562118400:0@lo:400:kworker.0 00010000:00010000:3.0:1777503007.185734:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777503008.209676:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777503009.233753:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503010.257740:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503011.281826:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777503011.921929:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777503011.921934:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777503011.921938:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777503011.921944:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9134 00000100:00080000:2.0:1777503011.921948:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777503011.921968:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777503011.922002:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f669500 x1863837562118912/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777503011.922020:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777503011.922024:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777503011.922027:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777503011.922031:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9134 00000100:00080000:2.0:1777503011.922035:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777503011.922039:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777503011.922065:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f669c00 x1863837562119040/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777503011.922136:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562118784:0@lo:400:kworker.0 00000100:00100000:2.0:1777503011.922216:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777503011.922309:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562118784 00000100:00100000:2.0:1777503011.922317:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562118784 00000100:00100000:2.0:1777503011.922356:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 489 00000100:00100000:2.0:1777503011.922365:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f698c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562118784:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777503011.922421:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f698c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562118784:12345-0@lo:400:kworker.0 Request processed in 56us (203us total) trans 0 rc 0/0 00000100:00100000:2.0:1777503011.922430:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 489 00000100:00100000:2.0:1777503011.922483:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562118784:0@lo:400:kworker.0 00000100:00100000:3.0:1777503011.922551:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f669500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562118912:0@lo:38:kworker.0 00000100:00100000:3.0:1777503011.922630:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503011.922656:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f669c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562119040:0@lo:38:kworker.0 00000100:00100000:3.0:1777503011.922685:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503011.922765:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562118912 00000100:00100000:3.0:1777503011.922769:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562118912 00000100:00100000:3.0:1777503011.922794:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562119040 00000100:00100000:3.0:1777503011.922796:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562119040 00000100:00100000:3.0:1777503011.922813:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16850 00000100:00100000:3.0:1777503011.922818:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc5ebf4a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562118912:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777503011.922833:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777503011.922873:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc5ebf4a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562118912:12345-0@lo:38:kworker.0 Request processed in 52us (243us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777503011.922881:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16850 00000100:00100000:3.0:1777503011.922900:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16851 00000100:00100000:3.0:1777503011.922906:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc5ebf6680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562119040:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777503011.922915:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777503011.922949:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc5ebf6680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562119040:12345-0@lo:38:kworker.0 Request processed in 41us (262us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777503011.922957:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16851 00000100:00080000:2.0:1777503011.923164:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777503011.923170:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777503011.923173:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f669500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562118912:0@lo:38:kworker.0 00000100:00080000:2.0:1777503011.923192:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777503011.923196:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777503011.923199:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f669c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562119040:0@lo:38:kworker.0 00010000:00010000:2.0:1777503012.305687:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777503013.329751:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503014.353717:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503015.377745:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777503016.401722:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777503017.041755:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777503017.041761:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777503017.041765:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777503017.041772:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9139 00000100:00080000:2.0:1777503017.041777:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777503017.041795:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777503017.041818:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f669500 x1863837562119296/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777503017.041840:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777503017.041843:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777503017.041845:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777503017.041849:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9139 00000100:00080000:2.0:1777503017.041852:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777503017.041856:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777503017.041871:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f66b480 x1863837562119424/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777503017.041900:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f669c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562119168:0@lo:400:kworker.0 00000100:00100000:2.0:1777503017.041981:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777503017.042075:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f669500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562119296:0@lo:38:kworker.0 00000100:00100000:3.0:1777503017.042115:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562119168 00000100:00100000:3.0:1777503017.042123:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562119168 00000100:00100000:2.0:1777503017.042156:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503017.042170:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 490 00000100:00100000:3.0:1777503017.042179:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f698050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562119168:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777503017.042186:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562119424:0@lo:38:kworker.0 00000100:00100000:2.0:1777503017.042219:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503017.042247:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f698050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562119168:12345-0@lo:400:kworker.0 Request processed in 69us (267us total) trans 0 rc 0/0 00000100:00100000:3.0:1777503017.042253:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 490 00000100:00100000:3.0:1777503017.042319:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562119296 00000100:00100000:3.0:1777503017.042322:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562119296 00000100:00100000:3.0:1777503017.042356:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562119424 00000100:00100000:3.0:1777503017.042359:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562119424 00000100:00100000:3.0:1777503017.042382:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16852 00000100:00100000:3.0:1777503017.042386:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f668700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562119296:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777503017.042399:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777503017.042425:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16853 00000100:00100000:3.0:1777503017.042431:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f668700 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562119296:12345-0@lo:38:kworker.0 Request processed in 42us (275us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777503017.042436:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f669f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562119424:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777503017.042442:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16852 00010000:02000000:0.0:1777503017.042450:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00080000:3.0:1777503017.042507:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777503017.042514:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777503017.042516:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f669f80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562119424:12345-0@lo:38:kworker.0 Request processed in 80us (295us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777503017.042520:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f669500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562119296:0@lo:38:kworker.0 00000100:00100000:0.0:1777503017.042533:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16853 00000100:00080000:3.0:1777503017.042544:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1777503017.042550:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:3.0:1777503017.042556:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562119424:0@lo:38:kworker.0 00000100:00100000:3.0:1777503017.042635:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f669c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562119168:0@lo:400:kworker.0 00010000:00010000:0.0:1777503017.425818:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503018.449727:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503019.473754:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777503020.497682:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777503021.521736:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1777503022.161836:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777503022.161842:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777503022.161847:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777503022.161855:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9144 00000100:00080000:3.0:1777503022.161860:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777503022.161868:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777503022.161897:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc5ebf5500 x1863837562119680/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1777503022.161911:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1777503022.161930:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1777503022.161933:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1777503022.161938:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9144 00000100:00080000:3.0:1777503022.161941:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1777503022.161957:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:3.0:1777503022.161975:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc5ebf5180 x1863837562119808/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1777503022.162008:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562119552:0@lo:400:kworker.0 00000100:00100000:3.0:1777503022.162074:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503022.162155:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562119680:0@lo:38:kworker.0 00000100:00100000:3.0:1777503022.162189:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503022.162216:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc5ebf5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562119808:0@lo:38:kworker.0 00000100:00100000:3.0:1777503022.162245:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503022.162289:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562119552 00000100:00100000:3.0:1777503022.162296:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562119552 00000100:00100000:3.0:1777503022.162344:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 491 00000100:00100000:3.0:1777503022.162358:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f698450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562119552:12345-0@lo:400:kworker.0 00000100:00100000:3.0:1777503022.162411:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f698450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562119552:12345-0@lo:400:kworker.0 Request processed in 58us (338us total) trans 0 rc 0/0 00000100:00100000:3.0:1777503022.162420:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 491 00000100:00100000:0.0:1777503022.162435:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562119680 00000100:00100000:0.0:1777503022.162440:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562119680 00000100:00100000:0.0:1777503022.162464:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562119808 00000100:00100000:0.0:1777503022.162467:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562119808 00000100:00100000:0.0:1777503022.162485:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16854 00000100:00100000:3.0:1777503022.162485:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562119552:0@lo:400:kworker.0 00000100:00100000:0.0:1777503022.162494:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc5ebf6a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562119680:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777503022.162553:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16855 00000100:00100000:3.0:1777503022.162560:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc5ebf4e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562119808:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777503022.162565:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:3.0:1777503022.162573:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:0.0:1777503022.162631:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc5ebf6a00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562119680:12345-0@lo:38:kworker.0 Request processed in 135us (442us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777503022.162637:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16854 00000100:00100000:3.0:1777503022.162642:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc5ebf4e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562119808:12345-0@lo:38:kworker.0 Request processed in 79us (395us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777503022.162650:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16855 00000100:00080000:0.0:1777503022.162696:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777503022.162701:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777503022.162704:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562119680:0@lo:38:kworker.0 00000100:00080000:0.0:1777503022.162714:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777503022.162718:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777503022.162721:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc5ebf5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562119808:0@lo:38:kworker.0 00010000:00010000:3.0:1777503022.545711:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503023.569739:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777503024.593677:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777503025.617711:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503026.641715:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777503027.281845:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777503027.281850:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777503027.281854:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777503027.281861:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9149 00000100:00080000:2.0:1777503027.281865:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777503027.281871:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777503027.281891:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f66ad80 x1863837562120064/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777503027.281918:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777503027.281921:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777503027.281923:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777503027.281926:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9149 00000100:00080000:2.0:1777503027.281929:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1777503027.281932:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777503027.281944:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f669c00 x1863837562120192/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777503027.281970:0:3309:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562119936:0@lo:400:kworker.0 00000100:00100000:2.0:1777503027.282027:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777503027.282111:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562119936 00000100:00100000:2.0:1777503027.282120:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562119936 00000100:00100000:2.0:1777503027.282180:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 492 00000100:00100000:2.0:1777503027.282188:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f69ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562119936:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777503027.282224:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562120064:0@lo:38:kworker.0 00000100:00100000:2.0:1777503027.282242:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f69ac50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3309:x1863837562119936:12345-0@lo:400:kworker.0 Request processed in 53us (213us total) trans 0 rc 0/0 00000100:00100000:2.0:1777503027.282251:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 492 00000100:00100000:0.0:1777503027.282268:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777503027.282295:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f669c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562120192:0@lo:38:kworker.0 00000100:00100000:2.0:1777503027.282310:0:3309:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d37a34a7-24bb-490e-b7a8-76d369c5604f:3309:1863837562119936:0@lo:400:kworker.0 00000100:00100000:0.0:1777503027.282328:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777503027.282364:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562120064 00000100:00100000:2.0:1777503027.282368:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562120064 00000100:00100000:2.0:1777503027.282394:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562120192 00000100:00100000:2.0:1777503027.282397:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562120192 00000100:00100000:2.0:1777503027.282421:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16856 00000100:00100000:2.0:1777503027.282428:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493fb100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562120064:12345-0@lo:38:kworker.0 00000100:00100000:0.0:1777503027.282434:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16857 00000100:00100000:0.0:1777503027.282442:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493f8000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562120192:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777503027.282446:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:0.0:1777503027.282455:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777503027.282491:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493fb100 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562120064:12345-0@lo:38:kworker.0 Request processed in 56us (218us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777503027.282502:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493f8000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562120192:12345-0@lo:38:kworker.0 Request processed in 57us (174us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777503027.282502:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16856 00000100:00100000:0.0:1777503027.282513:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16857 00000100:00080000:0.0:1777503027.282552:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777503027.282559:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777503027.282564:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f669c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562120192:0@lo:38:kworker.0 00000100:00080000:0.0:1777503027.282624:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777503027.282669:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777503027.282674:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562120064:0@lo:38:kworker.0 00010000:00010000:3.0:1777503027.665716:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777503028.689717:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777503029.713778:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503030.737813:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503031.761771:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777503032.401889:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777503032.401896:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777503032.401901:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777503032.401909:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9154 00000100:00080000:0.0:1777503032.401914:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1777503032.401920:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777503032.401941:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc493f8e00 x1863837562120448/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777503032.401960:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777503032.401963:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777503032.401966:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777503032.401970:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9154 00000100:00080000:0.0:1777503032.401972:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1777503032.401976:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777503032.401988:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc493f8a80 x1863837562120576/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777503032.402018:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493f8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562120448:0@lo:38:kworker.0 00000100:00100000:0.0:1777503032.402080:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:0.0:1777503032.402134:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493f8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562120576:0@lo:38:kworker.0 00000100:00100000:0.0:1777503032.402193:0:205369:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562120448 00000100:00100000:0.0:1777503032.402208:0:205369:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562120448 00000100:00100000:3.0:1777503032.402231:0:3310:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562120320:0@lo:400:kworker.0 00000100:00100000:0.0:1777503032.402268:0:205369:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16858 00000100:00100000:0.0:1777503032.402276:0:205369:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493fb800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562120448:12345-0@lo:38:kworker.0 00010000:02000000:0.0:1777503032.402292:0:205369:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777503032.402297:0:3310:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503032.402362:0:186930:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562120320 00000100:00100000:3.0:1777503032.402366:0:186930:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562120320 00000100:00100000:3.0:1777503032.402397:0:186930:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 493 00000100:00100000:3.0:1777503032.402404:0:186930:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f699c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562120320:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777503032.402418:0:205369:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493fb800 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_005:0+-99:3308:x1863837562120448:12345-0@lo:38:kworker.0 Request processed in 140us (337us total) trans 0 rc -19/-19 00000100:00100000:0.0:1777503032.402427:0:205369:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16858 00000100:00100000:3.0:1777503032.402446:0:186930:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f699c50 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0001:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3310:x1863837562120320:12345-0@lo:400:kworker.0 Request processed in 42us (148us total) trans 0 rc 0/0 00000100:00100000:3.0:1777503032.402455:0:186930:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 493 00000100:00100000:0.0:1777503032.402471:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503032.402517:0:3310:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d37a34a7-24bb-490e-b7a8-76d369c5604f:3310:1863837562120320:0@lo:400:kworker.0 00000100:00080000:0.0:1777503032.402520:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777503032.402538:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777503032.402543:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493f8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562120448:0@lo:38:kworker.0 00000100:00100000:3.0:1777503032.402600:0:186936:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562120576 00000100:00100000:3.0:1777503032.402604:0:186936:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562120576 00000100:00100000:3.0:1777503032.402625:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16859 00000100:00100000:3.0:1777503032.402630:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493fa680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562120576:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777503032.402637:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777503032.402670:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493fa680 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562120576:12345-0@lo:38:kworker.0 Request processed in 39us (202us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777503032.402676:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16859 00000100:00080000:0.0:1777503032.402682:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777503032.402712:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777503032.402717:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493f8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562120576:0@lo:38:kworker.0 00010000:00010000:2.0:1777503032.785681:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777503033.809754:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503034.833721:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503035.857743:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777503036.881713:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1777503037.521773:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777503037.521780:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777503037.521784:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777503037.521792:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9159 00000100:00080000:2.0:1777503037.521811:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1777503037.521820:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777503037.521844:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f66b480 x1863837562120832/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1777503037.521875:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1777503037.521878:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1777503037.521880:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1777503037.521884:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9159 00000100:00080000:2.0:1777503037.521887:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1777503037.521890:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:2.0:1777503037.521904:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfd4f669500 x1863837562120960/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1777503037.521928:0:3311:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562120704:0@lo:400:kworker.0 00000100:00100000:2.0:1777503037.521995:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777503037.522084:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f66b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562120832:0@lo:38:kworker.0 00000100:00100000:2.0:1777503037.522120:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777503037.522149:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfd4f669500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562120960:0@lo:38:kworker.0 00000100:00100000:2.0:1777503037.522181:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777503037.522225:0:186931:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562120704 00000100:00100000:2.0:1777503037.522233:0:186931:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562120704 00000100:00100000:2.0:1777503037.522283:0:186931:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 494 00000100:00100000:2.0:1777503037.522293:0:186931:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f699050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562120704:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1777503037.522354:0:186931:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f699050 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3311:x1863837562120704:12345-0@lo:400:kworker.0 Request processed in 63us (361us total) trans 0 rc 0/0 00000100:00100000:2.0:1777503037.522362:0:186931:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 494 00000100:00100000:0.0:1777503037.522379:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562120832 00000100:00100000:0.0:1777503037.522383:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562120832 00000100:00100000:0.0:1777503037.522414:0:187571:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562120960 00000100:00100000:0.0:1777503037.522417:0:187571:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562120960 00000100:00100000:2.0:1777503037.522435:0:3311:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:d37a34a7-24bb-490e-b7a8-76d369c5604f:3311:1863837562120704:0@lo:400:kworker.0 00000100:00100000:3.0:1777503037.522441:0:205372:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16860 00000100:00100000:0.0:1777503037.522451:0:187571:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16861 00000100:00100000:3.0:1777503037.522452:0:205372:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f668000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562120832:12345-0@lo:38:kworker.0 00000100:00100000:0.0:1777503037.522459:0:187571:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd4f669180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562120960:12345-0@lo:38:kworker.0 00010000:02000000:3.0:1777503037.522466:0:205372:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:0.0:1777503037.522468:0:187571:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:3.0:1777503037.522572:0:205372:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f668000 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_006:0+-99:3308:x1863837562120832:12345-0@lo:38:kworker.0 Request processed in 121us (451us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777503037.522628:0:205372:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16860 00000100:00080000:2.0:1777503037.522658:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1777503037.522672:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777503037.522677:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f66b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562120832:0@lo:38:kworker.0 00000100:00100000:0.0:1777503037.522692:0:187571:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd4f669180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:0+-99:3308:x1863837562120960:12345-0@lo:38:kworker.0 Request processed in 232us (510us total) trans 0 rc -19/-19 00000100:00080000:2.0:1777503037.522698:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1777503037.522699:0:187571:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16861 00000100:00080000:2.0:1777503037.522705:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:2.0:1777503037.522710:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfd4f669500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562120960:0@lo:38:kworker.0 00010000:00010000:0.0:1777503037.905753:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503038.929712:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503039.953752:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1777503040.977696:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777503042.001774:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1777503042.641766:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777503042.641830:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777503042.641834:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777503042.641843:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0000: connect to NID 0@lo last attempt 9164 00000100:00080000:0.0:1777503042.641850:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0000: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1777503042.641857:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0000: import ffff8dfd5361e000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777503042.641881:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc493f8e00 x1863837562121216/t0(0) o38->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1777503042.641898:0:173604:0:(pinger.c:229:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1777503042.641901:0:173604:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1777503042.641903:0:173604:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1777503042.641906:0:173604:0:(import.c:509:import_select_connection()) lustre-MDT0000-lwp-OST0001: connect to NID 0@lo last attempt 9164 00000100:00080000:0.0:1777503042.641909:0:173604:0:(import.c:588:import_select_connection()) lustre-MDT0000-lwp-OST0001: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1777503042.641913:0:173604:0:(import.c:623:import_select_connection()) lustre-MDT0000-lwp-OST0001: import ffff8dfd53619000 using connection 192.168.202.127@tcp/0@lo 00000100:00100000:0.0:1777503042.641925:0:173604:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dfc493f8700 x1863837562121344/t0(0) o38->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1777503042.641952:0:3312:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493f8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562121088:0@lo:400:kworker.0 00000100:00100000:0.0:1777503042.642010:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777503042.642202:0:186929:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562121088 00000100:00100000:2.0:1777503042.642208:0:186929:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562121088 00000100:00100000:2.0:1777503042.642253:0:186929:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 495 00000100:00100000:2.0:1777503042.642264:0:186929:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfd7f69a850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562121088:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1777503042.642291:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493f8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562121216:0@lo:38:kworker.0 00000100:00100000:2.0:1777503042.642324:0:186929:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfd7f69a850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:d37a34a7-24bb-490e-b7a8-76d369c5604f+13:3312:x1863837562121088:12345-0@lo:400:kworker.0 Request processed in 61us (312us total) trans 0 rc 0/0 00000100:00100000:0.0:1777503042.642329:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1777503042.642337:0:186929:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 495 00000100:00100000:0.0:1777503042.642352:0:3308:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562121344:0@lo:38:kworker.0 00000100:00100000:3.0:1777503042.642362:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562121216 00000100:00100000:3.0:1777503042.642369:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562121216 00000100:00100000:0.0:1777503042.642376:0:3308:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1777503042.642404:0:188648:0:(service.c:2282:ptlrpc_server_handle_req_in()) unwrap req x1863837562121344 00000100:00100000:3.0:1777503042.642409:0:188648:0:(service.c:2347:ptlrpc_server_handle_req_in()) got req x1863837562121344 00000100:00100000:0.0:1777503042.642427:0:3312:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493f8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:d37a34a7-24bb-490e-b7a8-76d369c5604f:3312:1863837562121088:0@lo:400:kworker.0 00000100:00100000:2.0:1777503042.642427:0:186936:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16862 00000100:00100000:3.0:1777503042.642446:0:188648:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 16863 00000100:00100000:2.0:1777503042.642450:0:186936:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc493f9880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562121216:12345-0@lo:38:kworker.0 00000100:00100000:3.0:1777503042.642457:0:188648:0:(service.c:2533:ptlrpc_server_handle_request()) Handling RPC req@ffff8dfc492e4e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562121344:12345-0@lo:38:kworker.0 00010000:02000000:2.0:1777503042.642468:0:186936:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00010000:02000000:3.0:1777503042.642470:0:188648:0:(ldlm_lib.c:1193:target_handle_connect()) lustre-MDT0000: Not available for connect from 0@lo (stopping) 00000100:00100000:2.0:1777503042.642524:0:186936:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc493f9880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_000:0+-99:3308:x1863837562121216:12345-0@lo:38:kworker.0 Request processed in 79us (195us total) trans 0 rc -19/-19 00000100:00100000:2.0:1777503042.642534:0:186936:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16862 00000100:00100000:3.0:1777503042.642534:0:188648:0:(service.c:2591:ptlrpc_server_handle_request()) Handled RPC req@ffff8dfc492e4e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_004:0+-99:3308:x1863837562121344:12345-0@lo:38:kworker.0 Request processed in 76us (155us total) trans 0 rc -19/-19 00000100:00100000:3.0:1777503042.642545:0:188648:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 16863 00000100:00080000:0.0:1777503042.642550:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd5361e000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777503042.642557:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777503042.642625:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493f8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0000_UUID:3308:1863837562121216:0@lo:38:kworker.0 00000100:00080000:0.0:1777503042.642641:0:3308:0:(import.c:65:import_set_state_nolock()) ffff8dfd53619000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1777503042.642644:0:3308:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 0@lo failed (-19) 00000100:00100000:0.0:1777503042.642647:0:3308:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff8dfc493f8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:lustre-MDT0000-lwp-OST0001_UUID:3308:1863837562121344:0@lo:38:kworker.0 00010000:00010000:3.0:1777503043.025790:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503044.049746:0:186924:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:02000400:2.0:1777503044.689693:0:221612:0:(genops.c:1873:obd_exports_barrier()) lustre-MDT0000 is waiting for obd_unlinked_exports more than 256 seconds. The obd refcount = 5. Is it stuck? 00000020:02000400:2.0:1777503044.693853:0:221612:0:(genops.c:1840:print_export_data()) lustre-MDT0000: UNLINKED ffff8dfd66b1b800 0775d2ab-558e-41f1-82c0-14244d133390 192.168.202.27@tcp 1 (0 0 0) 1 0 0 0: 0000000000000000 8589949329 stale:0 00010000:00010000:2.0:1777503045.073683:0:186925:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1777503046.097737:0:188362:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1777503047.121763:0:205387:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 6570 lines, 6570 kept, 0 dropped, 0 bad.