00010000:00010000:0.0F:1783953342.427209:0:5781:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0F:1783953343.462170:0:25300:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1783953344.475026:0:8361:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0F:1783953344.591182:0:3303:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.13@tcp (source: 12345-192.168.203.13@tcp) 00000100:00100000:3.0:1783953344.591271:0:3303:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.13@tcp (source: 12345-192.168.203.13@tcp) 00000100:00100000:0.0:1783953344.591480:0:71665:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1870607930902144 00000100:00100000:0.0:1783953344.591494:0:71665:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1870607930902144 00000100:00100000:0.0:1783953344.591565:0:71665:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.13@tcp, seq: 114 00000100:00100000:0.0:1783953344.591578:0:71665:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff970582ed1450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:901ec407-7db0-4817-91e0-6514bb3d22a5+10:2363:x1870607930902144:12345-192.168.203.13@tcp:400:kworker.0 00000100:00100000:0.0:1783953344.591684:0:71665:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff970582ed1450 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0000:901ec407-7db0-4817-91e0-6514bb3d22a5+10:2363:x1870607930902144:12345-192.168.203.13@tcp:400:kworker.0 Request processed in 108us (518us total) trans 0 rc 0/0 00000100:00100000:0.0:1783953344.591693:0:71665:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.13@tcp, seq: 114 00000100:00100000:2.0:1783953344.593123:0:71672:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1870607930902272 00000100:00100000:2.0:1783953344.593137:0:71672:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1870607930902272 00000100:00100000:2.0:1783953344.593224:0:71672:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.13@tcp, seq: 48 00000100:00100000:2.0:1783953344.593236:0:71672:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff970481e54a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:d01ce4e7-6009-4575-86bc-6575ed40d51a+14:2363:x1870607930902272:12345-192.168.203.13@tcp:400:kworker.0 00000100:00100000:2.0:1783953344.593321:0:71672:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff970481e54a80 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_002:d01ce4e7-6009-4575-86bc-6575ed40d51a+14:2363:x1870607930902272:12345-192.168.203.13@tcp:400:kworker.0 Request processed in 87us (2048us total) trans 0 rc 0/0 00000100:00100000:2.0:1783953344.593329:0:71672:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.13@tcp, seq: 48 00000100:00080000:3.0:1783953344.593748:0:5788:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00100000:1.0F:1783953344.927382:0:3311:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1783953344.927433:0:3311:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9705bf8adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3311:1870607950034304:0@lo:13:osp-pre-0-0.0 00000100:00100000:1.0:1783953344.927525:0:3311:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:1.0:1783953344.927607:0:6591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1870607950034304 00000100:00100000:1.0:1783953344.927616:0:6591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1870607950034304 00000100:00100000:1.0:1783953344.927688:0:6591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 971 00000100:00100000:1.0:1783953344.927699:0:6591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9705bf8ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3311:x1870607950034304:12345-0@lo:13:osp-pre-0-0.0 00002000:00000024:1.0:1783953344.927748:0:6591:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 21037056 pending 0 free 3859808256 avail 3857711104 00000100:00100000:1.0:1783953344.927814:0:6591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9705bf8ac700 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3311:x1870607950034304:12345-0@lo:13:osp-pre-0-0.0 Request processed in 116us (288us total) trans 0 rc 0/0 00000100:00100000:1.0:1783953344.927822:0:6591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 971 00000100:00080000:1.0:1783953344.927864:0:5788:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0000 00000100:00100000:1.0:1783953344.927921:0:3311:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9705bf8adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:lustre-MDT0000-mdtlov_UUID:3311:1870607950034304:0@lo:13:osp-pre-0-0.0 00000100:00100000:2.0:1783953344.935008:0:3309:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff970481e56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3309:1870607950034432:0@lo:13:osp-pre-1-0.0 00000100:00100000:2.0:1783953344.935114:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1783953344.935341:0:6592:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1870607950034432 00000100:00100000:3.0:1783953344.935352:0:6592:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1870607950034432 00000100:00100000:3.0:1783953344.935432:0:6592:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 972 00000100:00100000:3.0:1783953344.935443:0:6592:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff97058b4e9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3309:x1870607950034432:12345-0@lo:13:osp-pre-1-0.0 00002000:00000024:3.0:1783953344.935495:0:6592:0:(ofd_obd.c:735:ofd_statfs()) blocks cached 0 granted 21430272 pending 0 free 3859808256 avail 3857711104 00000100:00100000:3.0:1783953344.935558:0:6592:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff97058b4e9880 pname:cluuid+ref:pid:xid:nid:opc:job ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:3309:x1870607950034432:12345-0@lo:13:osp-pre-1-0.0 Request processed in 116us (443us total) trans 0 rc 0/0 00000100:00100000:3.0:1783953344.935566:0:6592:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 972 00000100:00100000:2.0:1783953344.935900:0:3309:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff970481e56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-mdtlov_UUID:3309:1870607950034432:0@lo:13:osp-pre-1-0.0 00000100:00100000:3.0:1783953344.936672:0:3312:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff970481e56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3312:1870607950034816:0@lo:400:kworker.0 00000100:00100000:3.0:1783953344.936731:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:3.0:1783953344.936763:0:3312:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:3.0:1783953344.936769:0:3312:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff970481e57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3312:1870607950034688:0@lo:400:kworker.0 00000100:00100000:3.0:1783953344.936794:0:3312:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1783953344.936926:0:72245:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1870607950034816 00000100:00100000:2.0:1783953344.936933:0:72245:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1870607950034816 00000100:00100000:2.0:1783953344.936988:0:72245:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1870607950034688 00000100:00100000:2.0:1783953344.936991:0:72245:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1870607950034688 00000100:00100000:2.0:1783953344.937017:0:72245:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 49 00000100:00100000:2.0:1783953344.937026:0:72245:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff970481e54e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0001_UUID+5:3312:x1870607950034816:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1783953344.937082:0:72245:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff970481e54e00 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0001_UUID+5:3312:x1870607950034816:12345-0@lo:400:kworker.0 Request processed in 56us (350us total) trans 0 rc 0/0 00000100:00100000:2.0:1783953344.937090:0:72245:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 49 00000100:00100000:2.0:1783953344.937107:0:72245:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 50 00000100:00100000:2.0:1783953344.937113:0:72245:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff970481e55880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3312:x1870607950034688:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1783953344.937139:0:72245:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff970481e55880 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-OST0000_UUID+5:3312:x1870607950034688:12345-0@lo:400:kworker.0 Request processed in 27us (344us total) trans 0 rc 0/0 00000100:00100000:2.0:1783953344.937145:0:72245:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 50 00000100:00100000:2.0:1783953344.937173:0:3309:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff970481e56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3309:1870607950034944:0@lo:400:kworker.0 00000100:00100000:2.0:1783953344.937200:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1783953344.937236:0:3309:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1783953344.937242:0:3309:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff970481e55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:602030f9-e829-4b6c-9995-c24116796ca7:3309:1870607950034560:0@lo:400:kworker.0 00000100:00100000:2.0:1783953344.937271:0:3309:0:(events.c:340:request_in_callback()) peer: 12345-0@lo (source: 12345-0@lo) 00000100:00100000:2.0:1783953344.937313:0:3312:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff970481e57800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:3312:1870607950034688:0@lo:400:kworker.0 00000100:00100000:2.0:1783953344.937324:0:3312:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff970481e56300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:3312:1870607950034816:0@lo:400:kworker.0 00000100:00100000:2.0:1783953344.937348:0:72245:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1870607950034944 00000100:00100000:2.0:1783953344.937351:0:72245:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1870607950034944 00000100:00100000:2.0:1783953344.937370:0:72245:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 51 00000100:00100000:2.0:1783953344.937376:0:72245:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff97058b4e9180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0000_UUID+5:3309:x1870607950034944:12345-0@lo:400:kworker.0 00000100:00100000:2.0:1783953344.937410:0:72245:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff97058b4e9180 pname:cluuid+ref:pid:xid:nid:opc:job mdt00_003:lustre-MDT0000-lwp-MDT0000_UUID+5:3309:x1870607950034944:12345-0@lo:400:kworker.0 Request processed in 33us (209us total) trans 0 rc 0/0 00000100:00100000:2.0:1783953344.937417:0:72245:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 51 00000100:00100000:2.0:1783953344.937442:0:3309:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff970481e56d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:lustre-MDT0000-lwp-MDT0000_UUID:3309:1870607950034944:0@lo:400:kworker.0 00000100:00100000:0.0:1783953344.937684:0:71667:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1870607950034560 00000100:00100000:0.0:1783953344.937690:0:71667:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1870607950034560 00000100:00100000:0.0:1783953344.937722:0:71667:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-0@lo, seq: 115 00000100:00100000:0.0:1783953344.937729:0:71667:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff970582ed1850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:602030f9-e829-4b6c-9995-c24116796ca7+19:3309:x1870607950034560:12345-0@lo:400:kworker.0 00000100:00100000:0.0:1783953344.937770:0:71667:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff970582ed1850 pname:cluuid+ref:pid:xid:nid:opc:job ll_mgs_0002:602030f9-e829-4b6c-9995-c24116796ca7+19:3309:x1870607950034560:12345-0@lo:400:kworker.0 Request processed in 42us (497us total) trans 0 rc 0/0 00000100:00100000:0.0:1783953344.937776:0:71667:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-0@lo, seq: 115 00000100:00100000:0.0:1783953344.937814:0:3309:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff970481e55180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:602030f9-e829-4b6c-9995-c24116796ca7:3309:1870607950034560:0@lo:400:kworker.0 00000100:00080000:0.0:1783953344.937883:0:5788:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-MDT0000 00000100:00080000:0.0:1783953344.937897:0:5788:0:(pinger.c:500:ping_evictor_main()) evicting all exports of obd lustre-OST0001 00000001:02000400:2.0:1783953345.301808:0:72928:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: ost-pools test_25: @@@@@@ FAIL: mds1: import is not in FULL state after 50 00010000:00010000:2.0:1783953345.499906:0:25454:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1783953346.525029:0:5780:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 83 lines, 83 kept, 0 dropped, 0 bad.