00000100:00000040:0.0F:1767316160.183256:0:18733:0:(connection.c:75:cpu_latency_work()) work item of ffff9d1f871e0800 (cpu 0) has reached its deadline 4296075097, at 4296075136 00000100:00000040:1.0F:1767316160.183571:0:36535:0:(connection.c:75:cpu_latency_work()) work item of ffff9d1f871e0910 (cpu 0) has reached its deadline 4296075097, at 4296075136 00010000:00010000:1.0:1767316160.374837:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316161.405018:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0F:1767316162.423812:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316163.448960:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767316164.408756:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2492:1853163904524672:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316164.408896:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904524928:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316164.408933:0:2492:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1767316164.408939:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904524800:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316164.408966:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904525056:192.168.202.140@tcp:400:kworker.0 00010000:00080000:3.0:1767316164.424325:0:2492:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f890e5c00 x1853163904525056/t0(0) o400->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316180 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316164.424346:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904525056:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316164.424359:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2492:1853163904524672:192.168.202.140@tcp:400:kworker.0 00010000:00080000:3.0:1767316164.424369:0:2492:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f890e4e00 x1853163904524928/t0(0) o400->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316180 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316164.424378:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904524928:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316164.427982:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904524800:192.168.202.140@tcp:400:kworker.0 00010000:00010000:1.0:1767316164.474580:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316165.495608:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316166.519014:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316167.544733:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316168.569916:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1767316169.534633:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904525184:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316169.542278:0:2490:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1767316169.542309:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904525312:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316169.543705:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904525568:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316169.543864:0:2491:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1767316169.543873:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904525440:192.168.202.140@tcp:400:kworker.0 00010000:00080000:0.0:1767316169.548266:0:2491:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f918d3480 x1853163904525568/t0(0) o400->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316185 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316169.548290:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904525568:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316169.548627:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904525312:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316169.550868:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904525184:192.168.202.140@tcp:400:kworker.0 00010000:00080000:1.0:1767316169.550985:0:2491:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f918d3b80 x1853163904525440/t0(0) o400->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316185 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316169.550998:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904525440:192.168.202.140@tcp:400:kworker.0 00010000:00010000:1.0:1767316169.592652:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316170.615150:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316171.638904:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767316172.663868:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0F:1767316173.582862:0:42877:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 136: MDS to disconnect all OSPs first, then cleanup ldlm ========================================================== 20:09:32 (1767316172) 00010000:00010000:3.0:1767316173.689560:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767316174.659444:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904525824:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316174.659550:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904526080:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316174.659582:0:2493:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1767316174.659588:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904525696:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316174.659628:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904525952:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316174.663193:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904525696:192.168.202.140@tcp:400:kworker.0 00010000:00080000:3.0:1767316174.663217:0:2493:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f92f73800 x1853163904525952/t0(0) o400->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316190 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316174.663239:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904525952:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316174.663252:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904525824:192.168.202.140@tcp:400:kworker.0 00010000:00080000:3.0:1767316174.663262:0:2493:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f92f72680 x1853163904526080/t0(0) o400->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316190 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316174.663271:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904526080:192.168.202.140@tcp:400:kworker.0 00010000:00010000:1.0:1767316174.710876:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1767316175.363777:0:43013:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_136 needs > 2 MDTs 00010000:00010000:1.0:1767316175.737321:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316176.762646:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1767316177.395561:0:43165:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 137a: DNE: create under striped dir, fail MDT1 ========================================================== 20:09:35 (1767316175) 00010000:00010000:1.0:1767316177.783959:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316178.807553:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1767316179.020818:0:43302:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_137a needs >= 2 MDTs 00000100:00080000:0.0:1767316179.771210:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad95800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:02000000:0.0:1767316179.771218:0:36:0:(import.c:2080:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff9d1f98126000: disconnect after 23s idle 00000100:00100000:1.0:1767316179.772398:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904526336:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316179.772528:0:2491:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1767316179.772536:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2491:1853163904526208:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316179.783526:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2491:1853163904526208:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316179.783555:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904526336:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316179.785642:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904526464:192.168.202.140@tcp:9:kworker.0 00000100:00080000:1.0:1767316179.785772:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad93000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:02000000:1.0:1767316179.785778:0:36:0:(import.c:2080:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff9d1f98126000: disconnect after 23s idle 00000100:00100000:0.0:1767316179.785791:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904526592:192.168.202.140@tcp:9:kworker.0 00010000:00080000:0.0:1767316179.794792:0:2489:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f918d3b80 x1853163904526592/t0(0) o9->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316189 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316179.794814:0:2489:0:(import.c:1990:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff9d1f918d3b80 x1853163904526592/t0(0) o9->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316189 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316179.794822:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad93000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1767316179.794834:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904526592:192.168.202.140@tcp:9:kworker.0 00010000:00080000:0.0:1767316179.794851:0:2489:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f91032a00 x1853163904526464/t0(0) o9->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316189 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316179.794862:0:2489:0:(import.c:1990:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff9d1f91032a00 x1853163904526464/t0(0) o9->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316189 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316179.794879:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad95800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1767316179.794883:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904526464:192.168.202.140@tcp:9:kworker.0 00010000:00010000:1.0:1767316179.842407:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1767316180.776320:0:43454:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 137b: DNE: create under striped dir, fail MDT2 ========================================================== 20:09:39 (1767316179) 00010000:00010000:3.0:1767316180.857237:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316181.883125:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1767316182.635597:0:43590:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_137b needs >= 2 MDTs 00010000:00010000:0.0:1767316182.902904:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316183.933803:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1767316184.548590:0:43743:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 137c: DNE: create under striped dir, fail MDT1/MDT2 ========================================================== 20:09:43 (1767316183) 00000100:00100000:3.0:1767316184.887413:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2492:1853163904526720:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316184.887558:0:2492:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1767316184.887567:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904526848:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316184.892593:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904526848:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316184.892622:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2492:1853163904526720:192.168.202.140@tcp:400:kworker.0 00010000:00010000:3.0:1767316184.974343:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316185.978393:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1767316186.111025:0:43879:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_137c needs >= 2 MDTs 00010000:00010000:0.0:1767316187.004467:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316188.029751:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1767316188.245980:0:44031:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 200: Dropping one OBD_PING should not cause disconnect ========================================================== 20:09:46 (1767316186) 00010000:00010000:3.0:1767316189.047311:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1767316189.874009:0:44167:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_200 Need remote client 00000100:00100000:0.0:1767316190.012293:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904526976:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316190.012418:0:2490:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1767316190.012427:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904527104:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316190.027864:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904527104:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316190.038114:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904526976:192.168.202.140@tcp:400:kworker.0 00010000:00010000:0.0:1767316190.072146:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316191.095078:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1767316191.661071:0:44320:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 201: MDT umount cascading disconnects timeouts ========================================================== 20:09:50 (1767316190) 00010000:00010000:3.0:1767316192.124081:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767316193.151884:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1767316193.402357:0:44459:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_201 needs >= 2 MDTs 00010000:00010000:3.0:1767316194.166967:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767316195.128966:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2492:1853163904527232:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316195.129073:0:2492:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1767316195.129080:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904527360:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316195.135337:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904527360:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316195.139764:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2492:1853163904527232:192.168.202.140@tcp:400:kworker.0 00010000:00010000:1.0:1767316195.191496:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1767316195.526287:0:44612:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 202: pfl replay should recovery layout generation ========================================================== 20:09:53 (1767316193) 00010000:00010000:3.0:1767316196.220147:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1767316196.570447:0:44638:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:3.0:1767316196.570876:0:44638:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316196.570900:0:44638:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316196.570917:0:44638:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316196.570926:0:44638:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1767316196.570938:0:44638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:3.0:1767316196.570942:0:44638:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16490f1f 00000080:00010000:3.0:1767316196.570950:0:44638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16490f1f from it ffffb28141d6fb58 00010000:00010000:3.0:1767316196.570954:0:44638:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316196.570979:0:44638:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316196.571001:0:44638:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:3.0:1767316196.571019:0:44638:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=d202.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=13 00000080:00200000:3.0:1767316196.571024:0:44638:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=d202.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:3.0:1767316196.571039:0:44638:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d202.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316196.571094:0:44638:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b1649105a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 44638 initiator: MDT0 00010000:00010000:3.0:1767316196.571103:0:44638:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b1649105a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44638 initiator: MDT0 00010000:00010000:3.0:1767316196.571125:0:44638:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b1649105a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44638 initiator: MDT0 00000100:00100000:3.0:1767316196.571155:0:44638:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44638:1853163904527488:192.168.202.140@tcp:101:lfs.0 00000100:00100000:3.0:1767316196.571364:0:44638:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f98809000 going to sleep for 11 seconds 00000100:00100000:3.0:1767316196.582638:0:44638:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44638:1853163904527488:192.168.202.140@tcp:101:lfs.0 00010000:00010000:3.0:1767316196.582665:0:44638:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b1649105a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44638 initiator: MDT0 00010000:00010000:3.0:1767316196.582713:0:44638:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b1649105a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 44638 initiator: MDT0 00010000:00010000:3.0:1767316196.582722:0:44638:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b1649105a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 44638 initiator: MDT0 00010000:00010000:3.0:1767316196.582729:0:44638:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b1649105a lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 44638 initiator: MDT0 00010000:00010000:3.0:1767316196.582758:0:44638:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b1649105a lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 44638 initiator: MDT0 00010000:00010000:3.0:1767316196.582765:0:44638:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f8b384400 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:3.0:1767316196.582771:0:44638:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b1649105a lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 44638 initiator: MDT0 00010000:00010000:3.0:1767316196.582778:0:44638:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b1649105a lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 44638 initiator: MDT0 00010000:00010000:3.0:1767316196.582787:0:44638:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b1649105a lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 44638 initiator: MDT0 00010000:00010000:3.0:1767316196.582794:0:44638:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1767316196.582801:0:44638:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b1649105a lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 44638 initiator: MDT0 00010000:00010000:3.0:1767316196.582808:0:44638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b1649105a lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 44638 initiator: MDT0 00000002:00100000:3.0:1767316196.582834:0:44638:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d1f92f72680 x1853163904527488/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/584 e 0 to 0 dl 1767316212 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1767316196.582851:0:44638:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1767316196.582892:0:44638:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316196.582906:0:44638:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00000080:00010000:3.0:1767316196.582914:0:44638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16490f1f from it ffffb28141d6f940 00010000:00010000:3.0:1767316196.582918:0:44638:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316196.582932:0:44638:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316196.583015:0:44638:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:3.0:1767316196.583029:0:44638:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316196.583036:0:44638:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316196.583045:0:44638:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316196.583051:0:44638:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1767316196.583056:0:44638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:3.0:1767316196.583061:0:44638:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16490f1f 00000080:00010000:3.0:1767316196.583066:0:44638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16490f1f from it ffffb28141d6fbe8 00010000:00010000:3.0:1767316196.583069:0:44638:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316196.583076:0:44638:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316196.583084:0:44638:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:3.0:1767316196.583093:0:44638:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d202.replay-single, flags=16392 00000080:00200008:3.0:1767316196.583131:0:44638:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1767316196.583136:0:44638:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:3.0:1767316196.583142:0:44638:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316196.583148:0:44638:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316196.583163:0:44638:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316196.583170:0:44638:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1767316196.583174:0:44638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:3.0:1767316196.583177:0:44638:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16490f1f 00000080:00010000:3.0:1767316196.583180:0:44638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16490f1f from it ffffb28141d6fcb8 00010000:00010000:3.0:1767316196.583184:0:44638:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316196.583191:0:44638:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00000080:00200008:2.0:1767316196.588285:0:44638:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1767316196.588298:0:44638:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316196.588326:0:44638:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316196.588348:0:44638:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.588366:0:44638:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316196.588387:0:44638:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316196.588395:0:44638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316196.588398:0:44638:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16490f1f 00000080:00010000:2.0:1767316196.588405:0:44638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16490f1f from it ffffb28141d6fcb8 00010000:00010000:2.0:1767316196.588409:0:44638:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.588417:0:44638:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316196.588582:0:44638:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316196.588592:0:44638:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316196.588598:0:44638:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.588606:0:44638:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316196.588618:0:44638:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316196.588622:0:44638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316196.588625:0:44638:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16490f1f 00000080:00010000:2.0:1767316196.588629:0:44638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16490f1f from it ffffb28141d6fbb0 00010000:00010000:2.0:1767316196.588633:0:44638:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.588640:0:44638:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316196.588649:0:44638:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 44 00000080:00200008:2.0:1767316196.588658:0:44638:0:(file.c:1001:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), flags 300000 00000080:00200080:2.0:1767316196.588932:0:44638:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c03066f0 arg=7fff81c8cb50 00000080:00200000:2.0:1767316196.588948:0:44638:0:(dir.c:483:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) name=d202.replay-single stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:2.0:1767316196.588971:0:44638:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.589032:0:44638:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.589039:0:44638:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316196.589042:0:44638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989d3000/0xa4df836b16490f1f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c8afb expref: -99 pid: 40424 timeout: 0 lvb_type: 0 00000020:00100000:2.0:1767316196.589065:0:44638:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316196.589079:0:44638:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e5f80 x1853163904527744/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 568/600 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316196.589130:0:44638:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5f80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44638:1853163904527744:192.168.202.140@tcp:36:lfs.0 00000100:00100000:2.0:1767316196.589238:0:44638:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f93cb4240 going to sleep for 11 seconds 00000100:00100000:2.0:1767316196.596817:0:44638:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5f80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44638:1853163904527744:192.168.202.140@tcp:36:lfs.0 00000080:00200000:2.0:1767316196.596864:0:44638:0:(llite_lib.c:3217:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8) 00000080:00200000:2.0:1767316196.596872:0:44638:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x467:0x0], updating i_size 512 00000080:00200008:2.0:1767316196.596905:0:44638:0:(file.c:424:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), flags 300000 00000080:00200000:1.0:1767316196.614927:0:44639:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316196.614953:0:44639:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316196.615013:0:44639:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 44639 initiator: MDT0 00010000:00010000:1.0:1767316196.615025:0:44639:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 44639 initiator: MDT0 00010000:00010000:1.0:1767316196.615034:0:44639:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 44639 initiator: MDT0 00000100:00100000:1.0:1767316196.615057:0:44639:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44639:1853163904527872:192.168.202.140@tcp:101:lfs.0 00000100:00100000:1.0:1767316196.615215:0:44639:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a1a6840 going to sleep for 11 seconds 00000100:00100000:1.0:1767316196.635669:0:44639:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44639:1853163904527872:192.168.202.140@tcp:101:lfs.0 00010000:00010000:1.0:1767316196.635730:0:44639:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316196.635761:0:44639:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316196.635773:0:44639:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316196.635782:0:44639:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316196.635794:0:44639:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316196.635801:0:44639:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316196.635809:0:44639:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000002:00100000:1.0:1767316196.635820:0:44639:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f918d1c00 x1853163904527872/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316212 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1767316196.635833:0:44639:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316196.635845:0:44639:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1767316196.635851:0:44639:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1767316196.635875:0:44639:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:1.0:1767316196.635881:0:44639:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491061 00000080:00010000:1.0:1767316196.635886:0:44639:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316196.635893:0:44639:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491061 from it ffffb28141d57b58 00010000:00010000:1.0:1767316196.635896:0:44639:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316196.635905:0:44639:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316196.635927:0:44639:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:1.0:1767316196.635944:0:44639:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=d202.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=8 00000080:00200000:1.0:1767316196.635948:0:44639:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=d202.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:1.0:1767316196.635967:0:44639:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d202.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316196.636002:0:44639:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 44639 initiator: MDT0 00010000:00010000:1.0:1767316196.636008:0:44639:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44639 initiator: MDT0 00010000:00010000:1.0:1767316196.636017:0:44639:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44639 initiator: MDT0 00000100:00100000:1.0:1767316196.636034:0:44639:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44639:1853163904528000:192.168.202.140@tcp:101:lfs.0 00000100:00100000:1.0:1767316196.636102:0:44639:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a1a6840 going to sleep for 11 seconds 00000100:00100000:2.0:1767316196.645043:0:44639:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44639:1853163904528000:192.168.202.140@tcp:101:lfs.0 00010000:00010000:2.0:1767316196.645069:0:44639:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.645092:0:44639:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.645100:0:44639:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.645127:0:44639:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.645138:0:44639:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.645151:0:44639:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.645158:0:44639:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.645184:0:44639:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316196.645196:0:44639:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f918d1c00 x1853163904528000/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/584 e 0 to 0 dl 1767316212 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316196.645209:0:44639:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.645219:0:44639:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1127 (0 0) 00000080:00200000:2.0:1767316196.645226:0:44639:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316196.645248:0:44639:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x467:0x0], updating i_size 512 00000080:00200000:2.0:1767316196.645251:0:44639:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x467:0x0](ffff9d1f884742c8): rc = 0 00000080:00010000:2.0:1767316196.645257:0:44639:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) for lock 0xa4df836b16491068 00000080:00010000:2.0:1767316196.645268:0:44639:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) 00000080:00010000:2.0:1767316196.645272:0:44639:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491068 from it ffffb28141d57b20 00010000:00010000:2.0:1767316196.645276:0:44639:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.645284:0:44639:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200008:2.0:1767316196.645321:0:44639:0:(file.c:6252:ll_getattr_dentry()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316196.645328:0:44639:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8),name=d202.replay-single 00000002:00010000:2.0:1767316196.645348:0:44639:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x467:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316196.645356:0:44639:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.645389:0:44639:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316196.645396:0:44639:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316196.645402:0:44639:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) 00000080:00010000:2.0:1767316196.645406:0:44639:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) for lock 0xa4df836b16491068 00000080:00010000:2.0:1767316196.645410:0:44639:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491068 from it ffffb28141d57cb8 00010000:00010000:2.0:1767316196.645413:0:44639:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.645419:0:44639:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316196.645473:0:44639:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316196.645481:0:44639:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316196.645488:0:44639:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.645495:0:44639:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316196.645500:0:44639:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316196.645504:0:44639:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316196.645506:0:44639:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491061 00000080:00010000:2.0:1767316196.645509:0:44639:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491061 from it ffffb28141d57bf8 00010000:00010000:2.0:1767316196.645512:0:44639:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316196.645527:0:44639:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316196.645535:0:44639:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316196.645542:0:44639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d202.replay-single, flags=267 00000080:00200000:2.0:1767316196.645548:0:44639:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), inode mode 41ed mask 44 00000080:00200008:2.0:1767316196.645557:0:44639:0:(file.c:1001:ll_file_open()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), flags 300000 00000002:00010000:2.0:1767316196.645570:0:44639:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x467:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:2.0:1767316196.645611:0:44639:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b1649106f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 44639 initiator: MDT0 00010000:00010000:2.0:1767316196.645618:0:44639:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b1649106f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44639 initiator: MDT0 00000020:00100000:2.0:1767316196.645818:0:44639:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1767316196.645825:0:44639:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e7480 x1853163904528256/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 672/131880 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1767316196.645835:0:44639:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b1649106f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44639 initiator: MDT0 00000100:00100000:3.0:1767316196.645917:0:44639:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7480 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44639:1853163904528256:192.168.202.140@tcp:101:lfs.0 00000100:00100000:3.0:1767316196.646087:0:44639:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f98809000 going to sleep for 11 seconds 00000100:00100000:3.0:1767316196.648428:0:44639:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7480 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44639:1853163904528256:192.168.202.140@tcp:101:lfs.0 00010000:00010000:3.0:1767316196.648450:0:44639:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b1649106f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44639 initiator: MDT0 00010000:00010000:3.0:1767316196.648462:0:44639:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b1649106f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 44639 initiator: MDT0 00010000:00010000:3.0:1767316196.648468:0:44639:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b1649106f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 44639 initiator: MDT0 00010000:00010000:3.0:1767316196.648475:0:44639:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b1649106f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 44639 initiator: MDT0 00010000:00010000:3.0:1767316196.648499:0:44639:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b1649106f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 44639 initiator: MDT0 00010000:00010000:3.0:1767316196.648504:0:44639:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f909f3000 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:3.0:1767316196.648509:0:44639:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b1649106f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 44639 initiator: MDT0 00010000:00010000:3.0:1767316196.648515:0:44639:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b1649106f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 44639 initiator: MDT0 00010000:00010000:3.0:1767316196.648521:0:44639:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b1649106f lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 44639 initiator: MDT0 00010000:00010000:3.0:1767316196.648527:0:44639:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1767316196.648533:0:44639:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b1649106f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 44639 initiator: MDT0 00010000:00010000:3.0:1767316196.648538:0:44639:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b1649106f lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 44639 initiator: MDT0 00000002:00100000:3.0:1767316196.648555:0:44639:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9d1f890e7480 x1853163904528256/t47244640408(47244640408) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 672/608 e 0 to 0 dl 1767316212 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1767316196.648566:0:44639:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e7480 x1853163904528256/t47244640408(47244640408) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 672/608 e 0 to 0 dl 1767316212 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1767316196.648577:0:44639:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1767316196.648595:0:44639:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x467:0x0], updating i_size 512 00000002:00100000:3.0:1767316196.648610:0:44639:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e7480 x1853163904528256/t47244640408(47244640408) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 672/608 e 0 to 0 dl 1767316212 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1767316196.648776:0:44639:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), inode mode 41ed mask 2 00000080:00200000:3.0:1767316196.648968:0:44639:0:(xattr.c:406:ll_xattr_set()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), xattr lov 00010000:00010000:3.0:1767316196.648991:0:44639:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316196.649043:0:44639:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316196.649050:0:44639:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1767316196.649053:0:44639:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc400/0xa4df836b16491068 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c91ca expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000020:00100000:3.0:1767316196.649072:0:44639:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1767316196.649079:0:44639:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f92f72680 x1853163904528512/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 680/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767316196.649112:0:44639:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44639:1853163904528512:192.168.202.140@tcp:36:lfs.0 00000100:00100000:3.0:1767316196.649199:0:44639:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f98809600 going to sleep for 11 seconds 00000100:00100000:3.0:1767316196.658799:0:44639:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44639:1853163904528512:192.168.202.140@tcp:36:lfs.0 00000080:00200008:3.0:1767316196.658859:0:44639:0:(file.c:424:ll_file_release()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), flags 300000 00000002:00100000:3.0:1767316196.658919:0:44639:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f890e7480 x1853163904528256/t47244640408(47244640408) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 672/608 e 0 to 0 dl 1767316212 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1767316196.658930:0:44639:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1767316196.658936:0:44639:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f92f71180 x1853163904528768/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/648 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1767316196.658959:0:44639:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44639:1853163904528768:192.168.202.140@tcp:35:lfs.0 00000100:00100000:3.0:1767316196.659026:0:44639:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f98809000 going to sleep for 11 seconds 00000100:00100000:3.0:1767316196.661136:0:44639:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44639:1853163904528768:192.168.202.140@tcp:35:lfs.0 00000002:00100000:3.0:1767316196.661179:0:44639:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f890e7480 x1853163904528256/t47244640408(47244640408) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 672/608 e 0 to 0 dl 1767316212 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1767316197.239036:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1767316197.556021:0:44649:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1767316197.556040:0:44649:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316197.556063:0:44649:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316197.556080:0:44649:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316197.556094:0:44649:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316197.556120:0:44649:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316197.556128:0:44649:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316197.556132:0:44649:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491061 00000080:00010000:0.0:1767316197.556139:0:44649:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491061 from it ffffb28141c47cb8 00010000:00010000:0.0:1767316197.556143:0:44649:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316197.556152:0:44649:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200008:0.0:1767316197.556733:0:44649:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1767316197.556740:0:44649:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316197.556749:0:44649:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316197.556757:0:44649:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316197.556765:0:44649:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316197.556772:0:44649:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316197.556776:0:44649:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316197.556779:0:44649:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491061 00000080:00010000:0.0:1767316197.556782:0:44649:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491061 from it ffffb28141c47cb8 00010000:00010000:0.0:1767316197.556785:0:44649:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316197.556792:0:44649:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316197.556890:0:44649:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316197.556898:0:44649:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316197.556905:0:44649:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316197.556913:0:44649:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316197.556919:0:44649:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316197.556923:0:44649:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316197.556925:0:44649:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491061 00000080:00010000:0.0:1767316197.556929:0:44649:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491061 from it ffffb28141c47bb0 00010000:00010000:0.0:1767316197.556932:0:44649:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316197.556939:0:44649:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316197.556947:0:44649:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 44 00000080:00200008:0.0:1767316197.556954:0:44649:0:(file.c:1001:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), flags 100000 00000080:00200080:0.0:1767316197.557045:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.557059:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00800000:00000004:0.0:1767316197.557067:0:44649:0:(obd_class.h:1083:obd_statfs()) lustre-MDT0000-mdc-ffff9d1f98126000: age 1404, max_age 1444 00000100:00100000:0.0:1767316197.557122:0:44649:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44649:1853163904528896:192.168.202.140@tcp:41:lfs.0 00000100:00100000:0.0:1767316197.557220:0:44649:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8af11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316197.565351:0:44649:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44649:1853163904528896:192.168.202.140@tcp:41:lfs.0 00800000:00000004:0.0:1767316197.565401:0:44649:0:(obd_class.h:1109:obd_statfs()) lustre-MDT0000-mdc-ffff9d1f98126000: update ffff9d1f90e03520 cache blocks 17229/17271 objects 499431/499796 00000080:00200080:0.0:1767316197.565611:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565628:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565637:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565641:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565646:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565649:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565654:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565657:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565661:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565664:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565668:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565671:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565674:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565705:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565711:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565714:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565718:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565722:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565726:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565729:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565733:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565736:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565739:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565742:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565755:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565759:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565763:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565766:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565770:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565773:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565777:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565781:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565785:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565788:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565959:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565965:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565969:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565972:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565976:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565979:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565983:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565986:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565990:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.565993:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.565997:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566001:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566005:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566008:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566011:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566033:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566040:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566044:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566048:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566052:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566056:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566060:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566064:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566067:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566070:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566073:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566076:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566080:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566083:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566086:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566090:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566094:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566108:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566113:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566117:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566120:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566124:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566127:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566131:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566135:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566146:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566150:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566154:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566158:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566162:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566165:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566169:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566173:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566177:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566181:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566185:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566188:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566192:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566195:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566199:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566202:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566206:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566210:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566214:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566217:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566221:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566225:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566229:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566232:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566235:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566246:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566251:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566255:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566259:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566262:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566265:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566268:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566272:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566275:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566278:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566281:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566284:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566288:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566292:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566296:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566300:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566304:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566308:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566311:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566315:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566318:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566321:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566325:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566328:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566331:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566342:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566347:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566351:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00800000:00000080:0.0:1767316197.566354:0:44649:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00000080:00200080:0.0:1767316197.566359:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:0.0:1767316197.566365:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00020000:00000004:0.0:1767316197.566373:0:44649:0:(obd_class.h:1083:obd_statfs()) lustre-OST0000-osc-ffff9d1f98126000: age 1404, max_age 1444 00000100:00080000:0.0:1767316197.566388:0:44649:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1391 00000100:00080000:0.0:1767316197.566399:0:44649:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d1f98126000: import ffff9d1f8ad95800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767316197.566409:0:44649:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767316197.566434:0:44649:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033480 x1853163904529024/t0(0) o8->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767316197.567792:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904529024:192.168.202.140@tcp:8:lfs.0 00000100:00080000:0.0:1767316197.567967:0:44649:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9d1f91031f80 x1853163904529152/t0(0) o13->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316197.567979:0:44649:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8af11300 going to sleep for 0 seconds 00010000:00080000:0.0:1767316197.574333:0:2489:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f91033480 x1853163904529024/t0(0) o8->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 520/416 e 0 to 0 dl 1767316207 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316197.574349:0:2489:0:(client.c:3081:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9d1f98126000: committing for last_committed 8589934598 gen 6 00000100:00080000:0.0:1767316197.574358:0:2489:0:(import.c:1123:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff9d1f98126000: connect to target with instance 4 00000100:00080000:0.0:1767316197.574368:0:2489:0:(import.c:951:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff9d1f98126000: Resetting ns_connect_flags to server flags: 0xa0425af2e3440038 00000080:00000004:0.0:1767316197.574379:0:2489:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0425af2e3440038 -> 0xa0425af2e3440038 00000080:00080000:0.0:1767316197.574384:0:2489:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1767316197.574389:0:2489:0:(import.c:1204:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:0.0:1767316197.574392:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad95800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1767316197.574397:0:2489:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0425af2e3440038 -> 0xa0425af2e3440038 00000080:00080000:0.0:1767316197.574413:0:2489:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1767316197.574421:0:2489:0:(recover.c:189:ptlrpc_wake_delayed()) @@@ waking (set 0000000045b3373c): req@ffff9d1f91031f80 x1853163904529152/t0(0) o13->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316197.574443:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904529024:192.168.202.140@tcp:8:lfs.0 00000100:00100000:1.0:1767316197.574724:0:44649:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8af11300 going to sleep for 11 seconds 00010000:00080000:0.0:1767316197.585388:0:44649:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f91031f80 x1853163904529152/t0(0) o13->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:7/4 lens 224/368 e 0 to 0 dl 1767316213 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316197.585409:0:44649:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44649:1853163904529152:192.168.202.140@tcp:13:lfs.0 00020000:00000004:0.0:1767316197.585437:0:44649:0:(obd_class.h:1109:obd_statfs()) lustre-OST0000-osc-ffff9d1f98126000: update ffff9d1f8ad44df8 cache blocks 3675/3683 objects 117664/118634 00000080:00200080:0.0:1767316197.585573:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:0.0:1767316197.585592:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98085000 uarg=0000000000000000 00020000:00000004:0.0:1767316197.585601:0:44649:0:(obd_class.h:1083:obd_statfs()) lustre-OST0001-osc-ffff9d1f98126000: age 1404, max_age 1444 00000100:00080000:0.0:1767316197.585613:0:44649:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1391 00000100:00080000:0.0:1767316197.585626:0:44649:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d1f98126000: import ffff9d1f8ad93000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767316197.585638:0:44649:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767316197.585660:0:44649:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033480 x1853163904529280/t0(0) o8->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767316197.585724:0:44649:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9d1f91031f80 x1853163904529408/t0(0) o13->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316197.585733:0:44649:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8af11300 going to sleep for 0 seconds 00000100:00100000:1.0:1767316197.585748:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904529280:192.168.202.140@tcp:8:lfs.0 00010000:00080000:1.0:1767316197.588767:0:2489:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f91033480 x1853163904529280/t0(0) o8->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 520/416 e 0 to 0 dl 1767316207 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316197.588866:0:2489:0:(client.c:3081:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9d1f98126000: committing for last_committed 4294968904 gen 8 00000100:00080000:1.0:1767316197.588889:0:2489:0:(import.c:1123:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff9d1f98126000: connect to target with instance 2 00000100:00080000:1.0:1767316197.588899:0:2489:0:(import.c:951:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff9d1f98126000: Resetting ns_connect_flags to server flags: 0xa0425af2e3440038 00000080:00000004:1.0:1767316197.588908:0:2489:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0425af2e3440038 -> 0xa0425af2e3440038 00000080:00080000:1.0:1767316197.588914:0:2489:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1767316197.588918:0:2489:0:(import.c:1204:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1767316197.588922:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad93000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1767316197.588927:0:2489:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0425af2e3440038 -> 0xa0425af2e3440038 00000080:00080000:1.0:1767316197.588930:0:2489:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1767316197.588937:0:2489:0:(recover.c:189:ptlrpc_wake_delayed()) @@@ waking (set 0000000045b3373c): req@ffff9d1f91031f80 x1853163904529408/t0(0) o13->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316197.588960:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904529280:192.168.202.140@tcp:8:lfs.0 00000100:00100000:0.0:1767316197.589413:0:44649:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8af11300 going to sleep for 11 seconds 00010000:00080000:1.0:1767316197.595330:0:44649:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f91031f80 x1853163904529408/t0(0) o13->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:7/4 lens 224/368 e 0 to 0 dl 1767316213 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316197.595352:0:44649:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44649:1853163904529408:192.168.202.140@tcp:13:lfs.0 00020000:00000004:1.0:1767316197.595379:0:44649:0:(obd_class.h:1109:obd_statfs()) lustre-OST0001-osc-ffff9d1f98126000: update ffff9d1f8ad43520 cache blocks 3677/3683 objects 117728/118123 00000080:00200080:1.0:1767316197.595440:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.595456:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.596157:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.596169:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.596173:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.596178:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.596182:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.596186:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.596190:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.596193:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597191:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597357:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597369:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597375:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597380:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597384:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597389:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597393:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597397:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597405:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597410:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597413:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597416:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597419:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597423:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597426:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597430:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597433:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597436:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597439:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597443:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597446:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597449:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597452:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597456:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597483:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597489:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597492:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597496:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597499:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597503:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597507:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597510:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597513:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597517:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597520:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597523:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597526:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597529:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597533:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597537:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597540:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597544:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597547:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597551:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597554:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597558:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597561:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597564:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597568:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597579:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597584:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597588:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597591:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597594:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597598:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597602:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597606:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597609:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597612:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597616:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597619:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597622:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597625:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597629:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597632:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597635:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597638:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597642:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597645:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597650:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597653:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597657:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597661:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597665:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597706:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597715:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597720:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597723:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597727:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597730:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597733:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597737:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597740:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597744:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597747:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597751:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597754:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597758:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597761:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597765:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597769:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597772:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597775:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597779:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597782:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597786:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597790:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597793:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597797:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597809:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597813:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597817:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597819:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597823:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597827:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597830:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597833:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597837:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597840:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597844:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597847:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597850:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597853:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200080:1.0:1767316197.597856:0:44649:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffeeec2a1c0 00020000:00000080:1.0:1767316197.597859:0:44649:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54400 uarg=0000000000000000 00000080:00200008:1.0:1767316197.597877:0:44649:0:(file.c:424:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), flags 100000 00000080:00200000:1.0:1767316197.656159:0:44656:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316197.656187:0:44656:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316197.656204:0:44656:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.656220:0:44656:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.656228:0:44656:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1767316197.656236:0:44656:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316197.656245:0:44656:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491061 00000080:00010000:1.0:1767316197.656252:0:44656:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491061 from it ffffb28141d57bd0 00010000:00010000:1.0:1767316197.656257:0:44656:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.656266:0:44656:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.656280:0:44656:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:1.0:1767316197.656292:0:44656:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=fsa-oleg240-client.virtnet, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=1536 00000080:00200000:1.0:1767316197.656296:0:44656:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316197.656302:0:44656:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316197.656309:0:44656:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.656316:0:44656:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.656322:0:44656:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1767316197.656326:0:44656:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316197.656329:0:44656:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491061 00000080:00010000:1.0:1767316197.656331:0:44656:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491061 from it ffffb28141d57cd0 00010000:00010000:1.0:1767316197.656334:0:44656:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.656341:0:44656:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.656348:0:44656:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 3 00000080:00200000:1.0:1767316197.656359:0:44656:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316197.656366:0:44656:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316197.656373:0:44656:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.656380:0:44656:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.656386:0:44656:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1767316197.656390:0:44656:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316197.656392:0:44656:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491061 00000080:00010000:1.0:1767316197.656396:0:44656:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491061 from it ffffb28141d57db0 00010000:00010000:1.0:1767316197.656399:0:44656:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.656407:0:44656:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.656415:0:44656:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 3 00000080:00200000:1.0:1767316197.656420:0:44656:0:(namei.c:2065:ll_create_nd()) VFS Op:name=fsa-oleg240-client.virtnet, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=33188, excl=1 00000080:00200000:1.0:1767316197.656423:0:44656:0:(namei.c:2020:ll_mknod()) VFS Op:name=fsa-oleg240-client.virtnet, dir=[0x200000007:0x1:0x0](ffff9d1f90125348) mode 100644 dev 0 00010000:00010000:1.0:1767316197.656450:0:44656:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.656501:0:44656:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.656508:0:44656:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1767316197.656510:0:44656:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491061 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c91ae expref: -99 pid: 44639 timeout: 0 lvb_type: 0 00000020:00100000:1.0:1767316197.656546:0:44656:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1767316197.656559:0:44656:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f918d0000 x1853163904529664/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 528/600 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767316197.656583:0:44656:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job mcreate:45d767c3-7963-411a-be8f-bcbeb1c06075:44656:1853163904529664:192.168.202.140@tcp:36:mcreate.0 00000100:00100000:1.0:1767316197.656675:0:44656:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a1a6d80 going to sleep for 11 seconds 00000100:00100000:1.0:1767316197.662982:0:44656:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job mcreate:45d767c3-7963-411a-be8f-bcbeb1c06075:44656:1853163904529664:192.168.202.140@tcp:36:mcreate.0 00000080:00200000:1.0:1767316197.663349:0:44656:0:(llite_lib.c:3217:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x468:0x0](ffff9d1f885d8908) 00000080:00200000:1.0:1767316197.663393:0:44656:0:(namei.c:2073:ll_create_nd()) VFS Op:name=fsa-oleg240-client.virtnet, unhashed 1 00000080:00200000:1.0:1767316197.683495:0:44659:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316197.683523:0:44659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316197.683587:0:44659:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 44659 initiator: MDT0 00010000:00010000:1.0:1767316197.683598:0:44659:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 44659 initiator: MDT0 00010000:00010000:1.0:1767316197.683608:0:44659:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 44659 initiator: MDT0 00000100:00100000:1.0:1767316197.683634:0:44659:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:44659:1853163904529792:192.168.202.140@tcp:101:rm.0 00000100:00100000:1.0:1767316197.683760:0:44659:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a1a6d80 going to sleep for 11 seconds 00000100:00100000:1.0:1767316197.691911:0:44659:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:44659:1853163904529792:192.168.202.140@tcp:101:rm.0 00010000:00010000:1.0:1767316197.691938:0:44659:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.691950:0:44659:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.691976:0:44659:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.692074:0:44659:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.692087:0:44659:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.692103:0:44659:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.692111:0:44659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00000002:00100000:1.0:1767316197.692122:0:44659:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f918d1f80 x1853163904529792/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316213 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1767316197.692136:0:44659:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.692147:0:44659:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1767316197.692154:0:44659:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1767316197.692179:0:44659:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:1.0:1767316197.692186:0:44659:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491076 00000080:00010000:1.0:1767316197.692191:0:44659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316197.692194:0:44659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491076 from it ffffb28141c17b58 00010000:00010000:1.0:1767316197.692197:0:44659:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.692212:0:44659:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.692237:0:44659:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:1.0:1767316197.692333:0:44659:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=fsa-oleg240-client.virtnet, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=8 00000080:00200000:1.0:1767316197.692338:0:44659:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=fsa-oleg240-client.virtnet, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:1.0:1767316197.692358:0:44659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: fsa-oleg240-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316197.692400:0:44659:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 44659 initiator: MDT0 00010000:00010000:1.0:1767316197.692407:0:44659:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44659 initiator: MDT0 00010000:00010000:1.0:1767316197.692415:0:44659:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44659 initiator: MDT0 00000100:00100000:1.0:1767316197.692434:0:44659:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:44659:1853163904529920:192.168.202.140@tcp:101:rm.0 00000100:00100000:1.0:1767316197.692522:0:44659:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a1a6d80 going to sleep for 11 seconds 00000100:00100000:1.0:1767316197.697577:0:44659:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:44659:1853163904529920:192.168.202.140@tcp:101:rm.0 00010000:00010000:1.0:1767316197.697605:0:44659:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.697629:0:44659:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.697638:0:44659:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.697667:0:44659:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.697708:0:44659:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.697721:0:44659:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.697729:0:44659:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.697736:0:44659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00000002:00100000:1.0:1767316197.697747:0:44659:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f918d1f80 x1853163904529920/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 600/584 e 0 to 0 dl 1767316213 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1767316197.697759:0:44659:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.697768:0:44659:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1128 (0 0) 00000080:00200000:1.0:1767316197.697775:0:44659:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1767316197.697795:0:44659:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x468:0x0], updating i_size 0 00000080:00200000:1.0:1767316197.697798:0:44659:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x468:0x0](ffff9d1f885d8908): rc = 0 00000080:00200000:1.0:1767316197.697812:0:44659:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x468:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:1.0:1767316197.697817:0:44659:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x468:0x0](ffff9d1f885d8908) for lock 0xa4df836b1649107d 00000080:00010000:1.0:1767316197.697827:0:44659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x468:0x0](ffff9d1f885d8908) 00000080:00010000:1.0:1767316197.697836:0:44659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649107d from it ffffb28141c17b20 00010000:00010000:1.0:1767316197.697839:0:44659:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.697847:0:44659:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00000080:00200008:1.0:1767316197.697881:0:44659:0:(file.c:6252:ll_getattr_dentry()) START file fsa-oleg240-client.virtnet:[0x200000bd1:0x468:0x0](ffff9d1f885d8908), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1767316197.697888:0:44659:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x468:0x0](ffff9d1f885d8908),name=fsa-oleg240-client.virtnet 00000002:00010000:1.0:1767316197.697905:0:44659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x468:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316197.697913:0:44659:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.697923:0:44659:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.697929:0:44659:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1767316197.697933:0:44659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x468:0x0](ffff9d1f885d8908) 00000080:00010000:1.0:1767316197.697936:0:44659:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x468:0x0](ffff9d1f885d8908) for lock 0xa4df836b1649107d 00000080:00010000:1.0:1767316197.697939:0:44659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649107d from it ffffb28141c17cb8 00010000:00010000:1.0:1767316197.697942:0:44659:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.697949:0:44659:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.698010:0:44659:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316197.698016:0:44659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316197.698022:0:44659:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.698034:0:44659:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.698041:0:44659:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1767316197.698044:0:44659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316197.698047:0:44659:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491076 00000080:00010000:1.0:1767316197.698050:0:44659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491076 from it ffffb28141c17c10 00010000:00010000:1.0:1767316197.698054:0:44659:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.698061:0:44659:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.698069:0:44659:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:1.0:1767316197.698077:0:44659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg240-client.virtnet, flags=0 00000080:00200000:1.0:1767316197.698082:0:44659:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316197.698086:0:44659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316197.698091:0:44659:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.698109:0:44659:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.698115:0:44659:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1767316197.698119:0:44659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316197.698121:0:44659:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491076 00000080:00010000:1.0:1767316197.698124:0:44659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491076 from it ffffb28141c17d68 00010000:00010000:1.0:1767316197.698127:0:44659:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.698139:0:44659:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.698147:0:44659:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 3 00000080:00200000:1.0:1767316197.698152:0:44659:0:(namei.c:2354:ll_unlink()) VFS Op:name=fsa-oleg240-client.virtnet, dir=[0x200000007:0x1:0x0](ffff9d1f90125348) 00010000:00010000:1.0:1767316197.698171:0:44659:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.698192:0:44659:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316197.698202:0:44659:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x468:0x0]: losing layout lock 00010000:00010000:1.0:1767316197.698232:0:44659:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.698239:0:44659:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.698244:0:44659:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1767316197.698247:0:44659:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649107d lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x468:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9225 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316197.698276:0:44659:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b16491076 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9209 expref: -99 pid: 44659 timeout: 0 lvb_type: 0 00000020:00100000:1.0:1767316197.698293:0:44659:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1767316197.698300:0:44659:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f918d1f80 x1853163904530176/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 512/648 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767316197.698332:0:44659:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:44659:1853163904530176:192.168.202.140@tcp:36:rm.0 00000100:00100000:1.0:1767316197.698412:0:44659:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a1a6d80 going to sleep for 11 seconds 00000100:00100000:1.0:1767316197.701994:0:44659:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:44659:1853163904530176:192.168.202.140@tcp:36:rm.0 00000080:00200000:1.0:1767316197.702056:0:44659:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x468:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1767316197.702069:0:44659:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x468:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1767316197.702077:0:44659:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x468:0x0](ffff9d1f885d8908) 00010000:00010000:3.0:1767316198.265598:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316199.287056:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1767316200.247140:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904530304:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316200.247174:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904530432:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316200.247258:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904530688:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316200.247269:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904530560:192.168.202.140@tcp:400:kworker.0 00010000:00080000:0.0:1767316200.252146:0:2490:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f918d1180 x1853163904530560/t0(0) o400->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316216 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316200.252184:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904530560:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316200.252196:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904530304:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316200.252793:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904530432:192.168.202.140@tcp:400:kworker.0 00010000:00080000:1.0:1767316200.252808:0:2491:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f918d3800 x1853163904530688/t0(0) o400->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316216 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316200.252817:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904530688:192.168.202.140@tcp:400:kworker.0 00010000:00010000:3.0:1767316200.310908:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1767316200.687996:0:44689:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000080:00200000:1.0:1767316200.713779:0:44690:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316200.713807:0:44690:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316200.713871:0:44690:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 44690 initiator: MDT0 00010000:00010000:1.0:1767316200.713881:0:44690:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 44690 initiator: MDT0 00010000:00010000:1.0:1767316200.713891:0:44690:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 44690 initiator: MDT0 00000100:00100000:1.0:1767316200.713927:0:44690:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job touch:45d767c3-7963-411a-be8f-bcbeb1c06075:44690:1853163904530816:192.168.202.140@tcp:101:touch.0 00000100:00100000:1.0:1767316200.714010:0:44690:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a1a6d80 going to sleep for 11 seconds 00000100:00100000:1.0:1767316200.722511:0:44690:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job touch:45d767c3-7963-411a-be8f-bcbeb1c06075:44690:1853163904530816:192.168.202.140@tcp:101:touch.0 00010000:00010000:1.0:1767316200.722537:0:44690:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.722549:0:44690:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.722560:0:44690:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.722571:0:44690:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.722582:0:44690:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.722589:0:44690:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.722597:0:44690:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000002:00100000:1.0:1767316200.722607:0:44690:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f918d3800 x1853163904530816/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316216 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1767316200.722621:0:44690:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.722650:0:44690:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1767316200.722657:0:44690:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1767316200.722717:0:44690:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:1.0:1767316200.722725:0:44690:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491084 00000080:00010000:1.0:1767316200.722731:0:44690:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316200.722734:0:44690:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491084 from it ffffb28141c9fbf8 00010000:00010000:1.0:1767316200.722737:0:44690:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.722757:0:44690:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316200.722784:0:44690:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:1.0:1767316200.722803:0:44690:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=d202.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=25 00000080:00200000:1.0:1767316200.722807:0:44690:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=d202.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=0 00000002:00010000:1.0:1767316200.722827:0:44690:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d202.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1767316200.722869:0:44690:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 44690 initiator: MDT0 00010000:00010000:1.0:1767316200.722877:0:44690:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44690 initiator: MDT0 00010000:00010000:1.0:1767316200.722885:0:44690:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44690 initiator: MDT0 00000100:00100000:1.0:1767316200.722906:0:44690:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job touch:45d767c3-7963-411a-be8f-bcbeb1c06075:44690:1853163904530944:192.168.202.140@tcp:101:touch.0 00000100:00100000:1.0:1767316200.722968:0:44690:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a1a6d80 going to sleep for 11 seconds 00000100:00100000:1.0:1767316200.730521:0:44690:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job touch:45d767c3-7963-411a-be8f-bcbeb1c06075:44690:1853163904530944:192.168.202.140@tcp:101:touch.0 00010000:00010000:1.0:1767316200.730563:0:44690:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c926b expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.730586:0:44690:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c926b expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.730593:0:44690:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c926b expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.730605:0:44690:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c926b expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.730616:0:44690:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c926b expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.730628:0:44690:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c926b expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.730636:0:44690:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c926b expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.730644:0:44690:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c926b expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000002:00100000:1.0:1767316200.730656:0:44690:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=10 disposition=b, status=0 req@ffff9d1f918d3800 x1853163904530944/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/584 e 0 to 0 dl 1767316216 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1767316200.730669:0:44690:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c926b expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.730975:0:44690:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1127 (0 0) 00000080:00200000:1.0:1767316200.730985:0:44690:0:(namei.c:955:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316200.731091:0:44690:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x467:0x0], updating i_size 11776 00000080:00200000:2.0:1767316200.731096:0:44690:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x467:0x0](ffff9d1f884742c8): rc = 0 00000080:00010000:2.0:1767316200.731103:0:44690:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) for lock 0xa4df836b1649108b 00000080:00010000:2.0:1767316200.731116:0:44690:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) 00000080:00010000:2.0:1767316200.731120:0:44690:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649108b from it ffffb28141c9fac8 00010000:00010000:2.0:1767316200.731123:0:44690:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c926b expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316200.731144:0:44690:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c926b expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316200.731195:0:44690:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), inode mode 41ed mask 1 00000080:00200000:2.0:1767316200.731216:0:44690:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), inode mode 41ed mask 3 00000080:00200000:2.0:1767316200.731222:0:44690:0:(namei.c:1354:ll_atomic_open()) VFS Op:name=f202.replay-single, dir=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), file ffff9d1f92ec9800, open_flags 8941, mode 81b6 opened 0 00000080:00200000:2.0:1767316200.731231:0:44690:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=f202.replay-single, dir=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), intent=open|creat 00000002:00010000:2.0:1767316200.731254:0:44690:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f202.replay-single,[0x200000bd1:0x469:0x0]) in obj [0x200000bd1:0x467:0x0], intent: open|creat flags 0104502 00010000:00010000:2.0:1767316200.731305:0:44690:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x467:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 44690 initiator: MDT0 00010000:00010000:2.0:1767316200.731313:0:44690:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x467:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44690 initiator: MDT0 00000020:00100000:2.0:1767316200.731324:0:44690:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1767316200.731332:0:44690:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e5500 x1853163904531200/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 688/66344 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1767316200.731340:0:44690:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x467:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44690 initiator: MDT0 00000100:00100000:2.0:1767316200.731360:0:44690:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5500 pname:cluuid:pid:xid:nid:opc:job touch:45d767c3-7963-411a-be8f-bcbeb1c06075:44690:1853163904531200:192.168.202.140@tcp:101:touch.0 00000100:00100000:2.0:1767316200.731493:0:44690:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f93cb4c00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316200.743605:0:44690:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5500 pname:cluuid:pid:xid:nid:opc:job touch:45d767c3-7963-411a-be8f-bcbeb1c06075:44690:1853163904531200:192.168.202.140@tcp:101:touch.0 00010000:00010000:2.0:1767316200.743640:0:44690:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x467:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316200.743732:0:44690:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316200.743740:0:44690:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316200.743759:0:44690:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316200.743771:0:44690:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316200.743784:0:44690:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316200.743792:0:44690:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316200.743799:0:44690:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316200.743811:0:44690:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d1f890e5500 x1853163904531200/t47244640413(47244640413) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 688/848 e 0 to 0 dl 1767316216 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1767316200.743824:0:44690:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e5500 x1853163904531200/t47244640413(47244640413) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 688/848 e 0 to 0 dl 1767316216 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316200.743838:0:44690:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316200.743863:0:44690:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316200.743874:0:44690:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1129 (0 0) 00000080:00200000:2.0:1767316200.743881:0:44690:0:(namei.c:955:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x467:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316200.743897:0:44690:0:(namei.c:1569:ll_create_it()) VFS Op:name=f202.replay-single, dir=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), intent=open|creat 00000080:00200000:2.0:1767316200.743934:0:44690:0:(llite_lib.c:3217:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x469:0x0](ffff9d1f88473248) 00000080:00200000:2.0:1767316200.744022:0:44690:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x469:0x0]: layout version change: 4294967294 -> 2 00000080:00010000:2.0:1767316200.744027:0:44690:0:(namei.c:1535:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x469:0x0](ffff9d1f88473248) 00000080:00010000:2.0:1767316200.744030:0:44690:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x469:0x0](ffff9d1f88473248) for lock 0xa4df836b16491092 00000080:00010000:2.0:1767316200.744041:0:44690:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x469:0x0](ffff9d1f88473248) 00000080:00200008:2.0:1767316200.744055:0:44690:0:(file.c:1001:ll_file_open()) START file f202.replay-single:[0x200000bd1:0x469:0x0](ffff9d1f88473248), flags 104501 00000002:00100000:2.0:1767316200.744083:0:44690:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e5500 x1853163904531200/t47244640413(47244640413) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/848 e 0 to 0 dl 1767316216 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1767316200.744119:0:44690:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491092 from it ffff9d1f88dd2e40 00010000:00010000:2.0:1767316200.744123:0:44690:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316200.744133:0:44690:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316200.744154:0:44690:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x469:0x0](ffff9d1f88473248), inode mode 81a4 mask 40 00000080:00200008:2.0:1767316200.744614:0:44690:0:(llite_lib.c:2314:ll_setattr_raw()) START file f202.replay-single:[0x200000bd1:0x469:0x0](ffff9d1f88473248) current size 0, valid attrs 20070, mode 9d1f, uid 1103756880, gid -19839, new size 18446744072481077783, atime 1767316200.000000000, mtime 1767316200.000000000, ctime 1767316200.000000000 00010000:00010000:2.0:1767316200.744640:0:44690:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316200.744670:0:44690:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x469:0x0]: losing layout lock 00010000:00010000:2.0:1767316200.744909:0:44690:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316200.744917:0:44690:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316200.744920:0:44690:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491092 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9287 expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316200.744941:0:44690:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316200.744948:0:44690:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e7100 x1853163904531456/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316200.744969:0:44690:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7100 pname:cluuid:pid:xid:nid:opc:job touch:45d767c3-7963-411a-be8f-bcbeb1c06075:44690:1853163904531456:192.168.202.140@tcp:36:touch.0 00000100:00100000:2.0:1767316200.745098:0:44690:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f93cb4c00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316200.758545:0:44690:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7100 pname:cluuid:pid:xid:nid:opc:job touch:45d767c3-7963-411a-be8f-bcbeb1c06075:44690:1853163904531456:192.168.202.140@tcp:36:touch.0 00000080:00200000:2.0:1767316200.758604:0:44690:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x469:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1767316200.758629:0:44690:0:(file.c:7031:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x469:0x0](ffff9d1f88473248) 00000002:00010000:2.0:1767316200.758640:0:44690:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x469:0x0]) in obj [0x200000bd1:0x469:0x0], intent: layout flags 0 00010000:00010000:2.0:1767316200.758738:0:44690:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 44690 initiator: MDT0 00010000:00010000:2.0:1767316200.758752:0:44690:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 44690 initiator: MDT0 00010000:00010000:2.0:1767316200.758762:0:44690:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 44690 initiator: MDT0 00000100:00100000:2.0:1767316200.758781:0:44690:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job touch:45d767c3-7963-411a-be8f-bcbeb1c06075:44690:1853163904531584:192.168.202.140@tcp:101:touch.0 00000100:00100000:2.0:1767316200.758854:0:44690:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f93cb4c00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316200.772604:0:44690:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job touch:45d767c3-7963-411a-be8f-bcbeb1c06075:44690:1853163904531584:192.168.202.140@tcp:101:touch.0 00010000:00010000:2.0:1767316200.772635:0:44690:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c929c expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316200.772651:0:44690:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c929c expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00000002:00100000:2.0:1767316200.772665:0:44690:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=400 disposition=1, status=0 req@ffff9d1f890e6a00 x1853163904531584/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 376/584 e 0 to 0 dl 1767316216 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316200.773115:0:44690:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c929c expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00000002:00010000:2.0:1767316200.773135:0:44690:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c929c expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316200.773148:0:44690:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 16 res 8589937617/1129 (0 0) 00000080:00200000:2.0:1767316200.773155:0:44690:0:(namei.c:955:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x469:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1767316200.773173:0:44690:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x469:0x0](ffff9d1f88473248) for lock 0xa4df836b16491099 00000080:00010000:2.0:1767316200.773183:0:44690:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491099 from it ffffb28141c9fa78 00010000:00010000:2.0:1767316200.773187:0:44690:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c929c expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316200.773197:0:44690:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c929c expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316200.773220:0:44690:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c929c expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316200.774649:0:44690:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c929c expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00000080:00010000:2.0:1767316200.774665:0:44690:0:(file.c:6914:ll_layout_lock_set()) ### file [0x200000bd1:0x469:0x0](00000000c407960c) being reconfigured ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c929c expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316200.774738:0:44690:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x469:0x0]: layout version change: 4294967294 -> 2 00010000:00010000:2.0:1767316200.775159:0:44690:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c929c expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316200.775170:0:44690:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c929c expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00020000:00200000:2.0:1767316200.775196:0:44690:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1767316200.775209:0:44690:0:(lov_io.c:860:lov_io_iter_init()) component[1] flags 0x0 00000100:00100000:3.0:1767316200.779311:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904531712:192.168.202.140@tcp:2:touch.0 00010000:00080000:3.0:1767316200.796633:0:2492:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f890e6a00 x1853163904531712/t0(0) o2->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 440/432 e 0 to 0 dl 1767316216 ref 1 fl Rpc:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1767316200.796712:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904531712:192.168.202.140@tcp:2:touch.0 00000080:00200000:2.0:1767316200.798022:0:44690:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x469:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1767316200.798044:0:44690:0:(llite_lib.c:2564:ll_setattr_raw()) COMPLETED file f202.replay-single:[0x200000bd1:0x469:0x0](ffff9d1f88473248) current size 0, valid attrs 201f0, mode 9d1f, uid 1103756880, gid -19839, new size 18446744072481077783, atime 1767316200.000000000, mtime 1767316200.000000000, ctime 1767316200.000000000, rc 0 00000080:00200008:2.0:1767316200.798088:0:44690:0:(file.c:424:ll_file_release()) START file f202.replay-single:[0x200000bd1:0x469:0x0](ffff9d1f88473248), flags 104001 00000002:00100000:2.0:1767316200.798171:0:44690:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f890e5500 x1853163904531200/t47244640413(47244640413) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/848 e 0 to 0 dl 1767316216 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1767316200.798187:0:44690:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1767316200.798192:0:44690:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e7800 x1853163904531968/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1767316200.798219:0:44690:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7800 pname:cluuid:pid:xid:nid:opc:job touch:45d767c3-7963-411a-be8f-bcbeb1c06075:44690:1853163904531968:192.168.202.140@tcp:35:touch.0 00000100:00100000:2.0:1767316200.798313:0:44690:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f93cb4c00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316200.804525:0:44690:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7800 pname:cluuid:pid:xid:nid:opc:job touch:45d767c3-7963-411a-be8f-bcbeb1c06075:44690:1853163904531968:192.168.202.140@tcp:35:touch.0 00000002:00100000:2.0:1767316200.804568:0:44690:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f890e5500 x1853163904531200/t47244640413(47244640413) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/848 e 0 to 0 dl 1767316216 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1767316200.842881:0:44692:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316200.842916:0:44692:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316200.842935:0:44692:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.842953:0:44692:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316200.842962:0:44692:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1767316200.842971:0:44692:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316200.842976:0:44692:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491084 00000080:00010000:1.0:1767316200.842983:0:44692:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491084 from it ffffb28141c97bf8 00010000:00010000:1.0:1767316200.842986:0:44692:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316200.842995:0:44692:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316200.843010:0:44692:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:1.0:1767316200.843018:0:44692:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d202.replay-single, flags=27 00000080:00200000:1.0:1767316200.843024:0:44692:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), inode mode 41ed mask 1 00000080:00200000:1.0:1767316200.843037:0:44692:0:(namei.c:1354:ll_atomic_open()) VFS Op:name=f202.replay-single, dir=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), file ffff9d1f9582e300, open_flags 18800, mode 0 opened 0 00000080:00200000:1.0:1767316200.843043:0:44692:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=f202.replay-single, dir=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), intent=open 00000002:00010000:1.0:1767316200.843058:0:44692:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f202.replay-single,[0x0:0x0:0x0]) in obj [0x200000bd1:0x467:0x0], intent: open flags 0304001 00010000:00010000:1.0:1767316200.843129:0:44692:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b164910a0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 44692 initiator: MDT0 00010000:00010000:1.0:1767316200.843142:0:44692:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b164910a0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44692 initiator: MDT0 00000020:00100000:1.0:1767316200.843156:0:44692:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1767316200.843169:0:44692:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f918d2d80 x1853163904532224/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1767316200.843178:0:44692:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b164910a0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44692 initiator: MDT0 00000100:00100000:1.0:1767316200.843203:0:44692:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44692:1853163904532224:192.168.202.140@tcp:101:lfs.0 00000100:00100000:1.0:1767316200.843395:0:44692:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a1a6d80 going to sleep for 11 seconds 00000100:00100000:1.0:1767316200.852496:0:44692:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44692:1853163904532224:192.168.202.140@tcp:101:lfs.0 00010000:00010000:1.0:1767316200.852519:0:44692:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b164910a0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44692 initiator: MDT0 00010000:00010000:1.0:1767316200.852533:0:44692:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b164910a0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 44692 initiator: MDT0 00010000:00010000:1.0:1767316200.852542:0:44692:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b164910a0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 44692 initiator: MDT0 00010000:00010000:1.0:1767316200.852549:0:44692:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b164910a0 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 44692 initiator: MDT0 00010000:00010000:1.0:1767316200.852558:0:44692:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b164910a0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 44692 initiator: MDT0 00010000:00010000:1.0:1767316200.852564:0:44692:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f93ccd800 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:1.0:1767316200.852569:0:44692:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b164910a0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 44692 initiator: MDT0 00010000:00010000:1.0:1767316200.852587:0:44692:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b164910a0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 44692 initiator: MDT0 00010000:00010000:1.0:1767316200.852594:0:44692:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b164910a0 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 44692 initiator: MDT0 00010000:00010000:1.0:1767316200.852600:0:44692:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1767316200.852607:0:44692:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b164910a0 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 44692 initiator: MDT0 00010000:00010000:1.0:1767316200.852614:0:44692:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b164910a0 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 44692 initiator: MDT0 00000002:00100000:1.0:1767316200.852633:0:44692:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=b, status=-20 req@ffff9d1f918d2d80 x1853163904532224/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/608 e 0 to 0 dl 1767316216 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1767316200.852647:0:44692:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000bd1:0x467:0x0] suppgids 0 -1: rc -20 00000080:00200000:0.0:1767316200.857089:0:44692:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316200.857124:0:44692:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316200.857144:0:44692:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316200.857161:0:44692:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316200.857169:0:44692:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316200.857179:0:44692:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316200.857183:0:44692:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491084 00000080:00010000:0.0:1767316200.857190:0:44692:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491084 from it ffffb28141c97bf8 00010000:00010000:0.0:1767316200.857195:0:44692:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316200.857204:0:44692:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316200.857222:0:44692:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316200.857229:0:44692:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d202.replay-single, flags=267 00000080:00200000:0.0:1767316200.857237:0:44692:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), inode mode 41ed mask 44 00000080:00200008:0.0:1767316200.857246:0:44692:0:(file.c:1001:ll_file_open()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), flags 304000 00000002:00010000:0.0:1767316200.857261:0:44692:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x467:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:0.0:1767316200.857313:0:44692:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164910a7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 44692 initiator: MDT0 00010000:00010000:0.0:1767316200.857334:0:44692:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164910a7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44692 initiator: MDT0 00000020:00100000:0.0:1767316200.857350:0:44692:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1767316200.857364:0:44692:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91030e00 x1853163904532480/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1767316200.857373:0:44692:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164910a7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44692 initiator: MDT0 00000100:00100000:0.0:1767316200.857400:0:44692:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030e00 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44692:1853163904532480:192.168.202.140@tcp:101:lfs.0 00000100:00100000:0.0:1767316200.857636:0:44692:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8af11e40 going to sleep for 11 seconds 00000100:00100000:2.0:1767316200.869450:0:44692:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030e00 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44692:1853163904532480:192.168.202.140@tcp:101:lfs.0 00010000:00010000:2.0:1767316200.869472:0:44692:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164910a7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 44692 initiator: MDT0 00010000:00010000:2.0:1767316200.869485:0:44692:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164910a7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 44692 initiator: MDT0 00010000:00010000:2.0:1767316200.869492:0:44692:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164910a7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 44692 initiator: MDT0 00010000:00010000:2.0:1767316200.869498:0:44692:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164910a7 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 44692 initiator: MDT0 00010000:00010000:2.0:1767316200.869505:0:44692:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164910a7 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 44692 initiator: MDT0 00010000:00010000:2.0:1767316200.869511:0:44692:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f93c84c00 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:2.0:1767316200.869515:0:44692:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164910a7 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 44692 initiator: MDT0 00010000:00010000:2.0:1767316200.869522:0:44692:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164910a7 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 44692 initiator: MDT0 00010000:00010000:2.0:1767316200.869528:0:44692:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164910a7 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 44692 initiator: MDT0 00010000:00010000:2.0:1767316200.869534:0:44692:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1767316200.869540:0:44692:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164910a7 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 44692 initiator: MDT0 00010000:00010000:2.0:1767316200.869562:0:44692:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164910a7 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 44692 initiator: MDT0 00000002:00100000:2.0:1767316200.869582:0:44692:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9d1f91030e00 x1853163904532480/t47244640416(47244640416) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 760/608 e 0 to 0 dl 1767316216 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1767316200.869595:0:44692:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f91030e00 x1853163904532480/t47244640416(47244640416) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 760/608 e 0 to 0 dl 1767316216 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1767316200.869608:0:44692:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316200.869629:0:44692:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x467:0x0], updating i_size 11776 00000002:00100000:2.0:1767316200.869645:0:44692:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f91030e00 x1853163904532480/t47244640416(47244640416) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 760/608 e 0 to 0 dl 1767316216 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1767316200.869835:0:44692:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8) cmd=c0086915 arg=26343d0 00000100:00100000:2.0:1767316200.869905:0:44692:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7b80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44692:1853163904532608:192.168.202.140@tcp:34:lfs.0 00000100:00100000:2.0:1767316200.869981:0:44692:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f93cb4c00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316200.875100:0:44692:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7b80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44692:1853163904532608:192.168.202.140@tcp:34:lfs.0 00000080:00200008:2.0:1767316200.875395:0:44692:0:(file.c:424:ll_file_release()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), flags 304000 00000002:00100000:2.0:1767316200.875460:0:44692:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f91030e00 x1853163904532480/t47244640416(47244640416) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 760/608 e 0 to 0 dl 1767316216 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1767316200.875471:0:44692:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1767316200.875477:0:44692:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e7b80 x1853163904532864/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1767316200.875497:0:44692:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7b80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44692:1853163904532864:192.168.202.140@tcp:35:lfs.0 00000100:00100000:2.0:1767316200.875566:0:44692:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f93cb4c00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316200.879366:0:44692:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7b80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:44692:1853163904532864:192.168.202.140@tcp:35:lfs.0 00000002:00100000:2.0:1767316200.879413:0:44692:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f91030e00 x1853163904532480/t47244640416(47244640416) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 760/608 e 0 to 0 dl 1767316216 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1767316201.335606:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316202.358970:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316203.382975:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316204.406965:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767316205.380083:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904532992:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316205.380206:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904533248:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316205.380244:0:2493:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1767316205.380253:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904533120:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316205.380277:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904533376:192.168.202.140@tcp:400:kworker.0 00010000:00080000:2.0:1767316205.388332:0:2493:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=92325) req@ffff9d1f92f71f80 x1853163904533376/t0(0) o400->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316221 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316205.388353:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904533376:192.168.202.140@tcp:400:kworker.0 00010000:00080000:2.0:1767316205.388368:0:2493:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=92325) req@ffff9d1f92f72a00 x1853163904533248/t0(0) o400->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316221 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316205.388375:0:2493:0:(client.c:3081:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9d1f98126000: committing for last_committed 12884901889 gen 6 00000100:00100000:2.0:1767316205.388393:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904533248:192.168.202.140@tcp:400:kworker.0 00010000:00010000:3.0:1767316205.431327:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316206.459194:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316207.479176:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316208.502973:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316209.527020:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316210.487124:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904533632:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316210.487229:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904533888:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316210.487264:0:2491:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1767316210.487271:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2491:1853163904533504:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316210.487302:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904533760:192.168.202.140@tcp:400:kworker.0 00010000:00080000:0.0:1767316210.498322:0:2491:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=92325) req@ffff9d1f918d1880 x1853163904533760/t0(0) o400->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316226 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316210.498342:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904533760:192.168.202.140@tcp:400:kworker.0 00010000:00080000:0.0:1767316210.498354:0:2491:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=92325) req@ffff9d1f918d1500 x1853163904533888/t0(0) o400->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316226 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316210.498363:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904533888:192.168.202.140@tcp:400:kworker.0 00010000:00010000:3.0:1767316210.551466:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316211.578387:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316212.598933:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316213.622963:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316214.647000:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316215.610385:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904534016:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316215.611891:0:2490:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1767316215.611929:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904534144:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316215.615573:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904534400:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316215.615719:0:2491:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1767316215.615727:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904534272:192.168.202.140@tcp:400:kworker.0 00010000:00080000:0.0:1767316215.622768:0:2491:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=92325) req@ffff9d1f91033480 x1853163904534272/t0(0) o400->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316231 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316215.622793:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904534272:192.168.202.140@tcp:400:kworker.0 00010000:00080000:0.0:1767316215.622806:0:2491:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=92325) req@ffff9d1f91032a00 x1853163904534400/t0(0) o400->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316231 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316215.622838:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904534400:192.168.202.140@tcp:400:kworker.0 00010000:00010000:1.0:1767316215.671810:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316216.694899:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316217.718971:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316218.743017:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316219.767151:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316220.728033:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904534528:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316220.728366:0:2490:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1767316220.728376:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904534656:192.168.202.140@tcp:400:kworker.0 00000100:00080000:0.0:1767316220.731753:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad95800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:02000000:0.0:1767316220.731762:0:36:0:(import.c:2080:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff9d1f98126000: disconnect after 20s idle 00000100:00080000:0.0:1767316220.746709:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad93000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:02000000:0.0:1767316220.746715:0:36:0:(import.c:2080:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff9d1f98126000: disconnect after 23s idle 00000100:00100000:1.0:1767316220.747323:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904534784:192.168.202.140@tcp:9:kworker.0 00000100:00100000:1.0:1767316220.747422:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904534912:192.168.202.140@tcp:9:kworker.0 00010000:00080000:0.0:1767316220.778518:0:2489:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=92325) req@ffff9d1f91031500 x1853163904534784/t0(0) o9->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316230 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316220.778540:0:2489:0:(import.c:1990:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff9d1f91031500 x1853163904534784/t0(0) o9->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316230 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316220.778548:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad95800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1767316220.778553:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904534784:192.168.202.140@tcp:9:kworker.0 00010000:00080000:0.0:1767316220.779743:0:2489:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=92325) req@ffff9d1f91031c00 x1853163904534912/t0(0) o9->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316230 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316220.779760:0:2489:0:(import.c:1990:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff9d1f91031c00 x1853163904534912/t0(0) o9->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316230 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316220.779767:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad93000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1767316220.779771:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904534912:192.168.202.140@tcp:9:kworker.0 00010000:00010000:3.0:1767316220.793210:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1767316221.751298:0:2493:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316205/real 1767316205] req@ffff9d1f92f73800 x1853163904533120/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316221 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316221.814861:0:2493:0:(client.c:2522:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d1f92f73800 x1853163904533120/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316221 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316221.814892:0:2493:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1767316221.814899:0:2493:0:(import.c:181:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff9d1f98126000: Connection to lustre-MDT0000 (at 192.168.202.140@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1767316221.865824:0:2493:0:(import.c:416:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00080000:0.0:1767316221.865941:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316221.865947:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00000400:2.0:1767316221.865947:0:2493:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316205/real 1767316205] req@ffff9d1f92f71180 x1853163904532992/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1767316221 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316221.865953:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316221.865963:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1251 00000100:00100000:2.0:1767316221.865984:0:2493:0:(client.c:2522:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d1f92f71180 x1853163904532992/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1767316221 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316221.865995:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767316221.866007:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1767316221.866008:0:2493:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1767316221.866017:0:2493:0:(import.c:185:ptlrpc_set_import_discon()) MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail 00000100:00100000:0.0:1767316221.866029:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031500 x1853163904535168/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767316221.866066:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904535168:192.168.202.140@tcp:38:kworker.0 00000100:00100000:0.0:1767316221.866268:0:2491:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00010000:00010000:3.0:1767316221.866314:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1767316221.895130:0:2493:0:(import.c:191:ptlrpc_set_import_discon()) import MGS@192.168.202.140@tcp for MGC192.168.202.140@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1767316221.895136:0:2493:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1767316221.895146:0:2493:0:(client.c:3559:ptlrpc_abort_inflight()) @@@ inflight req@ffff9d1f92f71180 x1853163904532992/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1767316221 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316221.895158:0:2493:0:(client.c:3559:ptlrpc_abort_inflight()) @@@ inflight req@ffff9d1f918d2d80 x1853163904533504/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 0 dl 1767316226 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316221.895168:0:2493:0:(client.c:3559:ptlrpc_abort_inflight()) @@@ inflight req@ffff9d1f91030e00 x1853163904534016/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 0 dl 1767316231 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316221.895179:0:2493:0:(client.c:3559:ptlrpc_abort_inflight()) @@@ inflight req@ffff9d1f91032a00 x1853163904534528/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 0 dl 1767316236 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316221.895224:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:0:1853163904535040:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316221.895323:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904534528:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316221.895332:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904534016:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316221.895371:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2491:1853163904533504:192.168.202.140@tcp:400:kworker.0 10000000:01000000:2.0:1767316221.895949:0:2493:0:(mgc_request.c:1165:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1767316221.895954:0:2493:0:(pinger.c:192:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1767316221.895956:0:2493:0:(mgc_request.c:1165:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1767316221.895958:0:2493:0:(import.c:416:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1767316221.896000:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316221.896006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767316221.896010:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316221.896016:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1251 00000100:00080000:0.0:1767316221.896029:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767316221.896060:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032680 x1853163904535296/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767316221.896161:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904535296:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767316221.897597:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904533120:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316221.897612:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904532992:192.168.202.140@tcp:400:kworker.0 00010000:00010000:3.0:1767316222.902985:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316223.927032:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316224.956876:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1767316225.910952:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316225.910963:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316225.910965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316225.910968:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316225.975011:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1767316226.935238:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316226.935253:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316226.935256:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316226.935259:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316226.998866:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1767316226.998897:0:2491:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316210/real 1767316210] req@ffff9d1f918d3100 x1853163904533632/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316226 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316226.998930:0:2491:0:(client.c:2522:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d1f918d3100 x1853163904533632/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316226 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316226.998939:0:2491:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9d1f980bb800 already not connected (conn 34, was 33): CONNECTING 00000100:00100000:0.0:1767316226.998959:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904533632:192.168.202.140@tcp:400:kworker.0 00010000:00010000:1.0:1767316228.023340:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316229.049322:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316230.073549:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767316231.096494:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1767316231.478850:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316221/real 1767316221] req@ffff9d1f91032680 x1853163904535296/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316231 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316231.478906:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032680 x1853163904535296/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316231 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316231.478916:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316221/real 1767316221] req@ffff9d1f91031500 x1853163904535168/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316231 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316231.478930:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031500 x1853163904535168/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316231 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316231.478960:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767316231.478995:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767316231.478999:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904535296:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767316231.479006:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767316231.479013:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767316231.479016:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904535168:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767316231.480141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316231.480149:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767316231.480153:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316231.480160:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1469 00000100:00080000:3.0:1767316231.480172:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1767316231.480178:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767316231.480223:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71180 x1853163904535424/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767316231.480255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316231.480259:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767316231.480272:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316231.480277:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1469 00000100:00080000:3.0:1767316231.480282:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1767316231.480285:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767316231.480295:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767316231.480308:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73800 x1853163904535552/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000080:00200000:3.0:1767316231.480359:0:36:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x469:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1767316231.480373:0:36:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:3.0:1767316231.480384:0:36:0:(lov_io.c:860:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:3.0:1767316231.480403:0:36:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x469:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1767316231.480438:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904535424:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767316231.480544:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904535552:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767316231.483985:0:2489:0:(import.c:1123:ptlrpc_connect_interpret()) MGC192.168.202.140@tcp: connect to target with instance 0 10000000:01000000:1.0:1767316231.484013:0:2489:0:(mgc_request.c:1165:mgc_import_event()) import event 0x808005 00000100:02000400:1.0:1767316231.484022:0:2489:0:(import.c:1275:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.202.140@tcp) after server handle changed from 0xcb3a17585d8c820a to 0xcb3a17585d8c92f7 00000100:00080000:1.0:1767316231.499030:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:1.0:1767316231.499046:0:2489:0:(import.c:1652:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.202.140@tcp; invalidating 00000100:00100000:0.0:1767316231.506186:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904535424:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767316231.509215:0:44960:0:(import.c:1584:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.202.140@tcp to MGS@192.168.202.140@tcp 00000100:00100000:1.0:1767316231.509222:0:44960:0:(import.c:321:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1767316231.509227:0:44960:0:(mgc_request.c:1165:mgc_import_event()) import event 0x808003 00010000:00010000:1.0:1767316231.509242:0:44960:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.140@tcp lock: ffff9d1f989cea00/0xa4df836b16490dc8 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0xcb3a17585d8c8640 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 10000000:00010000:1.0:1767316231.509252:0:44960:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.140@tcp lock: ffff9d1f989cea00/0xa4df836b16490dc8 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0xcb3a17585d8c8640 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 10000000:01000000:1.0:1767316231.509257:0:44960:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:1.0:1767316231.509341:0:44960:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1767316231.509351:0:44960:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.140@tcp lock: ffff9d1f989cea00/0xa4df836b16490dc8 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0xcb3a17585d8c8640 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316231.509365:0:44960:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.140@tcp lock: ffff9d1f989d3c00/0xa4df836b16490dcf lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xcb3a17585d8c8647 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 10000000:00010000:1.0:1767316231.509370:0:44960:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.140@tcp lock: ffff9d1f989d3c00/0xa4df836b16490dcf lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xcb3a17585d8c8647 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 10000000:01000000:1.0:1767316231.509375:0:44960:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:1.0:1767316231.513888:0:44960:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1767316231.513906:0:44960:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.140@tcp lock: ffff9d1f989d3c00/0xa4df836b16490dcf lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xcb3a17585d8c8647 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316231.513957:0:44960:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.140@tcp lock: ffff9d1f989d3600/0xa4df836b16491053 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xcb3a17585d8c916f expref: -99 pid: 7210 timeout: 0 lvb_type: 0 10000000:00010000:1.0:1767316231.513965:0:44960:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.140@tcp lock: ffff9d1f989d3600/0xa4df836b16491053 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xcb3a17585d8c916f expref: -99 pid: 7210 timeout: 0 lvb_type: 0 10000000:01000000:1.0:1767316231.513970:0:44960:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1767316231.517227:0:44960:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1767316231.517244:0:44960:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.140@tcp lock: ffff9d1f989d3600/0xa4df836b16491053 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xcb3a17585d8c916f expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316231.517275:0:44960:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.140@tcp lock: ffff9d1f989cec00/0xa4df836b16490d74 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0xcb3a17585d8c84a3 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 10000000:00010000:1.0:1767316231.517281:0:44960:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.140@tcp lock: ffff9d1f989cec00/0xa4df836b16490d74 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0xcb3a17585d8c84a3 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 10000000:01000000:1.0:1767316231.517286:0:44960:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:1.0:1767316231.521722:0:44960:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1767316231.521741:0:44960:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.140@tcp lock: ffff9d1f989cec00/0xa4df836b16490d74 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0xcb3a17585d8c84a3 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00000100:00080000:1.0:1767316231.521776:0:44960:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:1.0:1767316231.521782:0:44960:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from RECOVER to FULL 10000000:01000000:1.0:1767316231.521785:0:44960:0:(mgc_request.c:1165:mgc_import_event()) import event 0x808004 00000100:00080000:1.0:1767316231.521789:0:44960:0:(pinger.c:185:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:1.0:1767316231.521806:0:44960:0:(import.c:1722:ptlrpc_import_recovery_state_machine()) MGC192.168.202.140@tcp: Connection restored to 192.168.202.140@tcp (at 192.168.202.140@tcp) 00000100:00000400:1.0:1767316231.990825:0:2490:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316215/real 1767316215] req@ffff9d1f91031f80 x1853163904534144/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316231 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316231.990902:0:2490:0:(client.c:2522:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d1f91031f80 x1853163904534144/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316231 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767316231.990911:0:2490:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9d1f980bb800 already not connected (conn 35, was 33): CONNECTING 00000100:00100000:1.0:1767316231.990931:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904534144:192.168.202.140@tcp:400:kworker.0 00010000:00010000:1.0:1767316232.124023:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1767316233.143418:0:7210:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1767316233.143426:0:7210:0:(mgc_request.c:1812:mgc_process_log()) Process log lustre-cliir-ffff9d1f98126000 from 20 10000000:01000000:1.0:1767316233.143431:0:7210:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1767316233.143477:0:7210:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccda00/0xa4df836b164910ae lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316233.143487:0:7210:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccda00/0xa4df836b164910ae lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316233.143498:0:7210:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccda00/0xa4df836b164910ae lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00000100:00100000:1.0:1767316233.143526:0:7210:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904535680:192.168.202.140@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1767316233.143615:0:7210:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a189c00 going to sleep for 11 seconds 00010000:00010000:0.0:1767316233.143874:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316233.166874:0:7210:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904535680:192.168.202.140@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1767316233.166905:0:7210:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccda00/0xa4df836b164910ae lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c93ec expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316233.166918:0:7210:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccda00/0xa4df836b164910ae lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xcb3a17585d8c93ec expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00000100:00100000:1.0:1767316233.166998:0:7210:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904535808:192.168.202.140@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1767316233.167145:0:7210:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a189c00 going to sleep for 11 seconds 00000100:00100000:1.0:1767316233.206905:0:7210:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904535808:192.168.202.140@tcp:256:ll_cfg_requeue.0 00000020:00000080:1.0:1767316233.206972:0:7210:0:(obd_config.c:1389:class_process_config()) processing cmd: cf00f 10000000:01000000:1.0:1767316233.207099:0:7210:0:(mgc_request.c:1889:mgc_process_log()) MGC192.168.202.140@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1767316233.207109:0:7210:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccda00/0xa4df836b164910ae lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c93ec expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316233.207119:0:7210:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccda00/0xa4df836b164910ae lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c93ec expref: -99 pid: 7210 timeout: 0 lvb_type: 0 10000000:01000000:1.0:1767316233.207128:0:7210:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:1.0:1767316233.207130:0:7210:0:(mgc_request.c:1812:mgc_process_log()) Process log lustre-client-ffff9d1f98126000 from 55 10000000:01000000:1.0:1767316233.207133:0:7210:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1767316233.207169:0:7210:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccc400/0xa4df836b164910b5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316233.207175:0:7210:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccc400/0xa4df836b164910b5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316233.207184:0:7210:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccc400/0xa4df836b164910b5 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00000100:00100000:1.0:1767316233.207203:0:7210:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904535936:192.168.202.140@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1767316233.207277:0:7210:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a189c00 going to sleep for 11 seconds 00000100:00100000:1.0:1767316233.223408:0:7210:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904535936:192.168.202.140@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1767316233.223438:0:7210:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccc400/0xa4df836b164910b5 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c93f3 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316233.223451:0:7210:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccc400/0xa4df836b164910b5 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xcb3a17585d8c93f3 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00000100:00100000:1.0:1767316233.223547:0:7210:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536064:192.168.202.140@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1767316233.223620:0:7210:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a189c00 going to sleep for 11 seconds 00000100:00080000:2.0:1767316233.241117:0:44971:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9d1f980bb800 already not connected (conn 35, was 0): CONNECTING 00010000:00080000:2.0:1767316233.241149:0:44971:0:(ldlm_lib.c:85:import_set_conn()) imp ffff9d1f980bb800@lustre-MDT0000-mdc-ffff9d1f98126000: found existing conn 192.168.202.140@tcp, moved to head 00000100:00100000:1.0:1767316233.242883:0:7210:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536064:192.168.202.140@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1767316233.242946:0:7210:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536192:192.168.202.140@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1767316233.243041:0:7210:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a189c00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316233.247063:0:7210:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536192:192.168.202.140@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1767316233.250238:0:44972:0:(llog.c:854:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 54 00000020:01000000:2.0:1767316233.251195:0:7210:0:(obd_config.c:2043:class_config_parse_llog()) Processed log lustre-client gen 55-54 (rc=0) 10000000:01000000:2.0:1767316233.251221:0:7210:0:(mgc_request.c:1889:mgc_process_log()) MGC192.168.202.140@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1767316233.251231:0:7210:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccc400/0xa4df836b164910b5 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c93f3 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316233.251242:0:7210:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.140@tcp lock: ffff9d1f93ccc400/0xa4df836b164910b5 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c93f3 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 10000000:01000000:2.0:1767316233.251269:0:7210:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:2.0:1767316233.251272:0:7210:0:(mgc_request.c:1812:mgc_process_log()) Process log params-ffff9d1f98126000 from 7 10000000:01000000:2.0:1767316233.251274:0:7210:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1767316233.251313:0:7210:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f909f2a00/0xa4df836b164910bc lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316233.251320:0:7210:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.140@tcp lock: ffff9d1f909f2a00/0xa4df836b164910bc lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316233.251330:0:7210:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.140@tcp lock: ffff9d1f909f2a00/0xa4df836b164910bc lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1767316233.251358:0:7210:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536320:192.168.202.140@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1767316233.251457:0:7210:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f98809600 going to sleep for 11 seconds 00000100:00100000:2.0:1767316233.256655:0:7210:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536320:192.168.202.140@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1767316233.256729:0:7210:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.140@tcp lock: ffff9d1f909f2a00/0xa4df836b164910bc lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c93fa expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316233.256744:0:7210:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.140@tcp lock: ffff9d1f909f2a00/0xa4df836b164910bc lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xcb3a17585d8c93fa expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1767316233.256843:0:7210:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536448:192.168.202.140@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1767316233.256929:0:7210:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f98809600 going to sleep for 11 seconds 00000100:00100000:2.0:1767316233.261411:0:7210:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536448:192.168.202.140@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1767316233.261472:0:7210:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536576:192.168.202.140@tcp:503:ll_cfg_requeue.0 00000100:00100000:2.0:1767316233.261573:0:7210:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f98809600 going to sleep for 11 seconds 00000100:00100000:2.0:1767316233.277467:0:7210:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536576:192.168.202.140@tcp:503:ll_cfg_requeue.0 00000040:00080000:2.0:1767316233.286755:0:44973:0:(llog.c:854:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:3.0:1767316233.287109:0:7210:0:(obd_config.c:2043:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:3.0:1767316233.287295:0:7210:0:(mgc_request.c:1889:mgc_process_log()) MGC192.168.202.140@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1767316233.287307:0:7210:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f909f2a00/0xa4df836b164910bc lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c93fa expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316233.287320:0:7210:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.140@tcp lock: ffff9d1f909f2a00/0xa4df836b164910bc lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c93fa expref: -99 pid: 7210 timeout: 0 lvb_type: 0 10000000:01000000:3.0:1767316233.287330:0:7210:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:3.0:1767316233.287332:0:7210:0:(mgc_request.c:1812:mgc_process_log()) Process log lustre-sptlrpc-ffff9d1f980e18d8 from 1 10000000:01000000:3.0:1767316233.287335:0:7210:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:3.0:1767316233.287379:0:7210:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f8b385800/0xa4df836b164910c3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316233.287387:0:7210:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.140@tcp lock: ffff9d1f8b385800/0xa4df836b164910c3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316233.288036:0:7210:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.140@tcp lock: ffff9d1f8b385800/0xa4df836b164910c3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00000100:00100000:3.0:1767316233.288082:0:7210:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536704:192.168.202.140@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1767316233.288475:0:7210:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f877b8780 going to sleep for 11 seconds 00000100:00100000:0.0:1767316233.294023:0:7210:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536704:192.168.202.140@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1767316233.294059:0:7210:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.140@tcp lock: ffff9d1f8b385800/0xa4df836b164910c3 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9401 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316233.294072:0:7210:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.140@tcp lock: ffff9d1f8b385800/0xa4df836b164910c3 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xcb3a17585d8c9401 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1767316233.294170:0:7210:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536832:192.168.202.140@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1767316233.294382:0:7210:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8af11cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1767316233.319378:0:7210:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:7210:1853163904536832:192.168.202.140@tcp:501:ll_cfg_requeue.0 10000000:01000000:0.0:1767316233.319433:0:7210:0:(mgc_request.c:1889:mgc_process_log()) MGC192.168.202.140@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:0.0:1767316233.319445:0:7210:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f8b385800/0xa4df836b164910c3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9401 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316233.319456:0:7210:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.140@tcp lock: ffff9d1f8b385800/0xa4df836b164910c3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9401 expref: -99 pid: 7210 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316234.168155:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767316235.193004:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1767316236.087466:0:2490:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316220/real 1767316220] req@ffff9d1f91033480 x1853163904534656/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316236.087503:0:2490:0:(client.c:2522:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d1f91033480 x1853163904534656/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316236 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316236.087513:0:2490:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9d1f980bb800 already not connected (conn 35, was 33): CONNECTING 00000100:00100000:0.0:1767316236.087536:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904534656:192.168.202.140@tcp:400:kworker.0 00010000:00010000:1.0:1767316236.215035:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1767316236.598932:0:42605:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316236.598939:0:42605:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1767316236.598962:0:2490:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1767316236.598984:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904536960:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316236.611363:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904536960:192.168.202.140@tcp:400:kworker.0 00010000:00010000:0.0:1767316237.238966:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316238.265931:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767316239.288340:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316240.311018:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767316241.337231:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1767316241.626671:0:45062:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1767316241.626726:0:45062:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316241.626752:0:45062:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316241.626769:0:45062:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316241.626783:0:45062:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316241.626790:0:45062:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1767316241.626799:0:45062:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316241.626802:0:45062:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491084 00000080:00010000:1.0:1767316241.626817:0:45062:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491084 from it ffffb28140c87cb8 00010000:00010000:1.0:1767316241.626821:0:45062:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316241.626831:0:45062:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200008:1.0:1767316241.627389:0:45062:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1767316241.627396:0:45062:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316241.627404:0:45062:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316241.627412:0:45062:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316241.627422:0:45062:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316241.627429:0:45062:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1767316241.627434:0:45062:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316241.627437:0:45062:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491084 00000080:00010000:1.0:1767316241.627441:0:45062:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491084 from it ffffb28140c87cb8 00010000:00010000:1.0:1767316241.627444:0:45062:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316241.627452:0:45062:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316241.627548:0:45062:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316241.627556:0:45062:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316241.627563:0:45062:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316241.627579:0:45062:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316241.627586:0:45062:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1767316241.627590:0:45062:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316241.627593:0:45062:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491084 00000080:00010000:1.0:1767316241.627597:0:45062:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491084 from it ffffb28140c87bb0 00010000:00010000:1.0:1767316241.627600:0:45062:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316241.627606:0:45062:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316241.627613:0:45062:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 44 00000080:00200008:1.0:1767316241.627620:0:45062:0:(file.c:1001:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), flags 100000 00000080:00200080:1.0:1767316241.627753:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:1.0:1767316241.627766:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f92f54800 uarg=0000000000000000 00800000:00000004:1.0:1767316241.627772:0:45062:0:(obd_class.h:1083:obd_statfs()) lustre-MDT0000-mdc-ffff9d1f98126000: age 1445, max_age 1488 00000100:00080000:1.0:1767316241.627812:0:45062:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9d1f918d0e00 x1853163904537088/t0(0) o41->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316241.627823:0:45062:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f989decc0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316241.719354:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316241.719364:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1767316241.719430:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904537216:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316241.724236:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904537216:192.168.202.140@tcp:400:kworker.0 00010000:00010000:3.0:1767316242.358926:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316242.678824:0:45062:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f989decc0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316243.382880:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316243.702810:0:45062:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f989decc0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316244.406927:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316244.726816:0:45062:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f989decc0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316245.432357:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316245.750890:0:45062:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f989decc0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316246.454963:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316246.774794:0:45062:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f989decc0 going to sleep for 0 seconds 00000100:00100000:1.0:1767316246.838823:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316231/real 1767316231] req@ffff9d1f92f73800 x1853163904535552/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316246.838859:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73800 x1853163904535552/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316246 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767316246.838880:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767316246.838905:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767316246.838910:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904535552:192.168.202.140@tcp:38:kworker.0 00000100:00100000:3.0:1767316246.838947:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2492:1853163904537344:192.168.202.140@tcp:400:kworker.0 00000100:00080000:2.0:1767316246.839181:0:42605:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316246.839185:0:42605:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767316246.839187:0:42605:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767316246.839192:0:42605:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 0 00000100:00080000:2.0:1767316246.839200:0:42605:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767316246.839208:0:42605:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767316246.839221:0:42605:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f890e5880 x1853163904537472/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767316246.839401:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904537472:192.168.202.140@tcp:38:kworker.0 00000100:00100000:3.0:1767316246.843946:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2492:1853163904537344:192.168.202.140@tcp:400:kworker.0 00010000:00080000:0.0:1767316246.859023:0:2489:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f890e5880 x1853163904537472/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/416 e 0 to 0 dl 1767316256 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316246.859041:0:2489:0:(client.c:3081:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d1f98126000: committing for last_committed 47244640412 gen 2 00000100:00100000:0.0:1767316246.859081:0:2489:0:(client.c:3125:ptlrpc_free_committed()) @@@ stopping search req@ffff9d1f890e5500 x1853163904531200/t47244640413(47244640413) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/848 e 0 to 0 dl 1767316216 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1767316246.859099:0:2489:0:(import.c:1123:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to target with instance 12 00000100:00080000:0.0:1767316246.859116:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:0.0:1767316246.859124:0:2489:0:(import.c:1679:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1767316246.859126:0:2489:0:(client.c:3081:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d1f98126000: committing for last_committed 47244640412 gen 2 00000100:00100000:0.0:1767316246.859130:0:2489:0:(client.c:3125:ptlrpc_free_committed()) @@@ stopping search req@ffff9d1f890e5500 x1853163904531200/t47244640413(47244640413) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/848 e 0 to 0 dl 1767316216 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1767316246.859137:0:2489:0:(recover.c:53:ptlrpc_replay_next()) import ffff9d1f980bb800 from lustre-MDT0000_UUID committed 47244640412 last 0 00000100:00080000:0.0:1767316246.859143:0:2489:0:(client.c:3523:ptlrpc_replay_req()) @@@ REPLAY req@ffff9d1f890e5500 x1853163904531200/t47244640413(47244640413) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/848 e 0 to 0 dl 1767316216 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1767316246.859157:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904537472:192.168.202.140@tcp:38:kworker.0 00000100:00100000:0.0:1767316246.859181:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904531200:192.168.202.140@tcp:101:touch.0 00000100:00080000:0.0:1767316246.916976:0:2489:0:(client.c:3408:ptlrpc_replay_interpret()) @@@ got reply req@ffff9d1f890e5500 x1853163904531200/t47244640413(47244640413) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/608 e 0 to 0 dl 1767316262 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 00000002:00080000:0.0:1767316246.916998:0:2489:0:(mdc_request.c:714:mdc_replay_open()) @@@ updating close body with new fh req@ffff9d1f890e7800 x1853163904531968/t47244640415(47244640415) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/456 e 0 to 0 dl 1767316216 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000100:00020000:0.0:1767316246.917008:0:2489:0:(client.c:3418:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff9d1f890e5500 x1853163904531200/t47244640413(47244640413) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/608 e 0 to 0 dl 1767316262 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1767316246.947946:0:2489:0:(import.c:1679:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1767316246.947953:0:2489:0:(client.c:3081:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d1f98126000: committing for last_committed 47244640412 gen 2 00000100:00100000:0.0:1767316246.947965:0:2489:0:(client.c:3125:ptlrpc_free_committed()) @@@ stopping search req@ffff9d1f890e5500 x1853163904531200/t47244640413(47244640413) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/608 e 0 to 0 dl 1767316262 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1767316246.947972:0:2489:0:(recover.c:53:ptlrpc_replay_next()) import ffff9d1f980bb800 from lustre-MDT0000_UUID committed 47244640412 last 47244640413 00000100:00080000:0.0:1767316246.947979:0:2489:0:(client.c:3523:ptlrpc_replay_req()) @@@ REPLAY req@ffff9d1f890e7100 x1853163904531456/t47244640414(47244640414) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/456 e 0 to 0 dl 1767316216 ref 1 fl New:RQU/204/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316246.947990:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904531200:192.168.202.140@tcp:101:touch.0 00000100:00100000:0.0:1767316246.948010:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904531456:192.168.202.140@tcp:36:touch.0 00000100:00080000:0.0:1767316246.955525:0:2489:0:(client.c:3408:ptlrpc_replay_interpret()) @@@ got reply req@ffff9d1f890e7100 x1853163904531456/t47244640414(47244640414) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/456 e 0 to 0 dl 1767316262 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316246.955537:0:2489:0:(import.c:1679:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1767316246.955539:0:2489:0:(client.c:3081:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d1f98126000: committing for last_committed 47244640412 gen 2 00000100:00100000:0.0:1767316246.955551:0:2489:0:(client.c:3125:ptlrpc_free_committed()) @@@ stopping search req@ffff9d1f890e5500 x1853163904531200/t47244640413(47244640413) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/608 e 0 to 0 dl 1767316262 ref 1 fl Complete:RQU/604/0 rc 0/301 job:'touch.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1767316246.955556:0:2489:0:(recover.c:53:ptlrpc_replay_next()) import ffff9d1f980bb800 from lustre-MDT0000_UUID committed 47244640412 last 47244640414 00000100:00080000:0.0:1767316246.955561:0:2489:0:(client.c:3523:ptlrpc_replay_req()) @@@ REPLAY req@ffff9d1f890e7800 x1853163904531968/t47244640415(47244640415) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/456 e 0 to 0 dl 1767316216 ref 2 fl New:RQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1767316246.955666:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904531456:192.168.202.140@tcp:36:touch.0 00000100:00100000:0.0:1767316246.955722:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904531968:192.168.202.140@tcp:35:touch.0 00000100:00080000:0.0:1767316246.965945:0:2489:0:(client.c:3408:ptlrpc_replay_interpret()) @@@ got reply req@ffff9d1f890e7800 x1853163904531968/t47244640415(47244640415) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/456 e 0 to 0 dl 1767316262 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1767316246.965974:0:2489:0:(import.c:1679:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1767316246.965977:0:2489:0:(client.c:3081:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d1f98126000: committing for last_committed 47244640412 gen 2 00000100:00100000:0.0:1767316246.965984:0:2489:0:(client.c:3125:ptlrpc_free_committed()) @@@ stopping search req@ffff9d1f890e5500 x1853163904531200/t47244640413(47244640413) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/608 e 0 to 0 dl 1767316262 ref 1 fl Complete:RQU/604/0 rc 0/301 job:'touch.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1767316246.965992:0:2489:0:(recover.c:53:ptlrpc_replay_next()) import ffff9d1f980bb800 from lustre-MDT0000_UUID committed 47244640412 last 47244640415 00000100:00080000:0.0:1767316246.965997:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:1.0:1767316246.967050:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904531968:192.168.202.140@tcp:35:touch.0 00010000:00080000:0.0:1767316246.967098:0:45078:0:(ldlm_request.c:2714:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff9d1f98126000 to lustre-MDT0000_UUID@192.168.202.140@tcp 00010000:00010000:0.0:1767316246.967102:0:45078:0:(ldlm_request.c:2634:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff9d1f98126000 (29) 00010000:00010000:0.0:1767316246.967453:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc600/0xa4df836b16491084 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0xcb3a17585d8c924f expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316246.967627:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x469:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.967640:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b16491099 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x469:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c929c expref: -99 pid: 44690 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316246.967715:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd200/0xa4df836b1649108b lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0xcb3a17585d8c926b expref: -99 pid: 44690 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316246.967731:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x451:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.967738:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93d37a00/0xa4df836b16490f0a lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x451:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c8a99 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.967756:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x44e:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.967763:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b16490ee0 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c8a06 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.967855:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x44f:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.967877:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93d36e00/0xa4df836b16490eee lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c8a37 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.967893:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x44d:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.967900:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b16490ed2 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c89d5 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.967912:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x44b:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.967918:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84e00/0xa4df836b16490eb6 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c8973 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.967930:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x448:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.967935:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f92f9f200/0xa4df836b16490e8c lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x448:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c88e0 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.967945:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x44a:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.967950:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989ce200/0xa4df836b16490ea8 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c8942 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.967963:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x44c:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.967969:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f92f9ea00/0xa4df836b16490ec4 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c89a4 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.967980:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x447:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.967985:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f92f9f600/0xa4df836b16490e7e lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x447:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c88af expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.967995:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x446:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.968000:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f92f9e600/0xa4df836b16490e70 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x446:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c887e expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.968024:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x444:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.968031:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f92f9fc00/0xa4df836b16490e54 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x444:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c881c expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.968043:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x450:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.968049:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93d36800/0xa4df836b16490efc lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x450:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c8a68 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.968061:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x443:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.968067:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f92f9ee00/0xa4df836b16490e46 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x443:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c87eb expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.968078:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x43f:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.968083:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989cf200/0xa4df836b16490e0e lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c8727 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.968093:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x449:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.968098:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f92f9fe00/0xa4df836b16490e9a lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x449:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c8911 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.968109:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x442:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.968114:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f92f9f800/0xa4df836b16490e38 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x442:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c87ba expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.968126:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x445:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.968131:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f92f9e400/0xa4df836b16490e62 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x445:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c884d expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.968142:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x441:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.968156:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f92f9e000/0xa4df836b16490e2a lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x441:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c8789 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.968169:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x440:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.968174:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989cf600/0xa4df836b16490e1c lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x440:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c8758 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316246.968185:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x43e:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.968190:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989cf400/0xa4df836b16490e00 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c86f6 expref: -99 pid: 40310 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316246.968210:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f92f9e200/0xa4df836b16490df2 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0xcb3a17585d8c86c5 expref: -99 pid: 40310 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316246.968222:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x43b:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.968233:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b16490dac lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c8552 expref: -99 pid: 38577 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316246.968253:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85800/0xa4df836b16490da5 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0xcb3a17585d8c8536 expref: -99 pid: 38577 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316246.968406:0:45078:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x43a:0x0]: losing layout lock 00010000:00010000:0.0:1767316246.968420:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989cfe00/0xa4df836b16490d97 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0xcb3a17585d8c8513 expref: -99 pid: 38577 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316246.968437:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989ce400/0xa4df836b16490d90 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0xcb3a17585d8c84f7 expref: -99 pid: 38577 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316246.968453:0:45078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f989cee00/0xa4df836b16490d89 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0xcb3a17585d8c84db expref: -99 pid: 38577 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316246.968595:0:45078:0:(ldlm_request.c:2647:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff9d1f98126000 00000100:00080000:0.0:1767316246.969188:0:45078:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:0.0:1767316246.969642:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904537600:192.168.202.140@tcp:400:ldlm_lock_repla.0 00000100:00080000:2.0:1767316247.026970:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:2.0:1767316247.026993:0:2489:0:(client.c:3203:ptlrpc_resend_req()) @@@ going to resend req@ffff9d1f918d2d80 x1853163904537600/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 0 dl 1767316262 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316247.027006:0:2489:0:(client.c:3212:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff9d1f918d2d80 x1853163904537600/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 0 dl 1767316262 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316247.027012:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:2.0:1767316247.027875:0:2489:0:(import.c:1722:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff9d1f98126000: Connection restored to 192.168.202.140@tcp (at 192.168.202.140@tcp) 00000100:00080000:2.0:1767316247.027884:0:2489:0:(recover.c:189:ptlrpc_wake_delayed()) @@@ waking (set 000000000f5e8568): req@ffff9d1f918d0e00 x1853163904537088/t0(0) o41->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316247.027910:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904537600:192.168.202.140@tcp:400:ldlm_lock_repla.0 00000100:00100000:3.0:1767316247.028120:0:45062:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f989decc0 going to sleep for 11 seconds 00000100:00100000:3.0:1767316247.033538:0:45062:0:(client.c:3081:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d1f98126000: committing for last_committed 47244640415 gen 2 00000100:00100000:3.0:1767316247.033593:0:45062:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45062:1853163904537088:192.168.202.140@tcp:41:lfs.0 00800000:00000004:3.0:1767316247.033611:0:45062:0:(obd_class.h:1109:obd_statfs()) lustre-MDT0000-mdc-ffff9d1f98126000: update ffff9d1f90e03520 cache blocks 17229/17271 objects 499700/500067 00000080:00200080:3.0:1767316247.033923:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.033941:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.033950:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.033970:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.033976:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.033980:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.033984:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.033986:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034004:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034008:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034020:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034025:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034030:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034857:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034867:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034871:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034876:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034880:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034883:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034887:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034891:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034894:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034899:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034902:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034906:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034910:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034914:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034917:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034921:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034925:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034929:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034932:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034936:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034947:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034952:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034956:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034960:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034963:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034968:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034971:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034975:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034978:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034982:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034985:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034988:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034991:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.034994:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.034998:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035001:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035004:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035008:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035020:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035025:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035029:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035188:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035196:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035202:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035205:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035217:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035222:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035227:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035230:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035234:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035237:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035242:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035250:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035255:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035260:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035265:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035268:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035272:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035275:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035278:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035281:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035285:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035288:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035291:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035294:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035298:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035301:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035305:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035309:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035312:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035322:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035327:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035331:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035335:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035339:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035342:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035346:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035349:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035352:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035355:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035358:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035548:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035556:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035560:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035715:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035723:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035727:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035731:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035734:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035738:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035742:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035745:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035748:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035751:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035754:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035767:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035771:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035775:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035779:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035782:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035786:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035790:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035794:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035798:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035801:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035805:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035808:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035812:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035816:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035819:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035823:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035827:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035831:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035835:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00800000:00000080:3.0:1767316247.035839:0:45062:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00000080:00200080:3.0:1767316247.035844:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:3.0:1767316247.035849:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f99339400 uarg=0000000000000000 00020000:00000004:3.0:1767316247.035857:0:45062:0:(obd_class.h:1083:obd_statfs()) lustre-OST0000-osc-ffff9d1f98126000: age 1445, max_age 1493 00000100:00080000:3.0:1767316247.035872:0:45062:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1445 00000100:00080000:3.0:1767316247.035885:0:45062:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d1f98126000: import ffff9d1f8ad95800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767316247.035902:0:45062:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767316247.035927:0:45062:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70700 x1853163904537728/t0(0) o8->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316247.036003:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904537728:192.168.202.140@tcp:8:lfs.0 00000100:00080000:3.0:1767316247.036194:0:45062:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9d1f92f73480 x1853163904537856/t0(0) o13->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316247.036203:0:45062:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90afe000 going to sleep for 0 seconds 00010000:00080000:2.0:1767316247.046322:0:2489:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f92f70700 x1853163904537728/t0(0) o8->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 520/416 e 0 to 0 dl 1767316257 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316247.046340:0:2489:0:(client.c:3081:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9d1f98126000: committing for last_committed 12884901889 gen 7 00000100:00080000:2.0:1767316247.046350:0:2489:0:(import.c:1123:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff9d1f98126000: connect to target with instance 4 00000100:00080000:2.0:1767316247.046369:0:2489:0:(import.c:951:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff9d1f98126000: Resetting ns_connect_flags to server flags: 0xa0425af2e3440038 00000080:00000004:2.0:1767316247.046383:0:2489:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0425af2e3440038 -> 0xa0425af2e3440038 00000080:00080000:2.0:1767316247.046390:0:2489:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1767316247.046396:0:2489:0:(import.c:1204:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:2.0:1767316247.046399:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad95800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:2.0:1767316247.046405:0:2489:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0425af2e3440038 -> 0xa0425af2e3440038 00000080:00080000:2.0:1767316247.046408:0:2489:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1767316247.046416:0:2489:0:(recover.c:189:ptlrpc_wake_delayed()) @@@ waking (set 00000000f626f8b5): req@ffff9d1f92f73480 x1853163904537856/t0(0) o13->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316247.046440:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904537728:192.168.202.140@tcp:8:lfs.0 00000100:00100000:3.0:1767316247.047003:0:45062:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90afe000 going to sleep for 11 seconds 00010000:00080000:3.0:1767316247.051964:0:45062:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f92f73480 x1853163904537856/t0(0) o13->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:7/4 lens 224/368 e 0 to 0 dl 1767316263 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316247.051988:0:45062:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45062:1853163904537856:192.168.202.140@tcp:13:lfs.0 00020000:00000004:3.0:1767316247.052015:0:45062:0:(obd_class.h:1109:obd_statfs()) lustre-OST0000-osc-ffff9d1f98126000: update ffff9d1f8ad44df8 cache blocks 3675/3683 objects 117664/118634 00000080:00200080:2.0:1767316247.052287:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.052307:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00020000:00000004:2.0:1767316247.052314:0:45062:0:(obd_class.h:1083:obd_statfs()) lustre-OST0001-osc-ffff9d1f98126000: age 1445, max_age 1493 00000100:00080000:2.0:1767316247.052327:0:45062:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1445 00000100:00080000:2.0:1767316247.052338:0:45062:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d1f98126000: import ffff9d1f8ad93000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767316247.052349:0:45062:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767316247.052381:0:45062:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f890e4380 x1853163904537984/t0(0) o8->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767316247.052419:0:45062:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9d1f890e6d80 x1853163904538112/t0(0) o13->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316247.052444:0:45062:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1fc2173c00 going to sleep for 0 seconds 00000100:00100000:1.0:1767316247.052823:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904537984:192.168.202.140@tcp:8:lfs.0 00010000:00080000:2.0:1767316247.056426:0:2489:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f890e4380 x1853163904537984/t0(0) o8->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 520/416 e 0 to 0 dl 1767316257 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316247.056443:0:2489:0:(client.c:3081:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9d1f98126000: committing for last_committed 4294968904 gen 9 00000100:00080000:2.0:1767316247.056454:0:2489:0:(import.c:1123:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff9d1f98126000: connect to target with instance 2 00000100:00080000:2.0:1767316247.056465:0:2489:0:(import.c:951:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff9d1f98126000: Resetting ns_connect_flags to server flags: 0xa0425af2e3440038 00000080:00000004:2.0:1767316247.056475:0:2489:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0425af2e3440038 -> 0xa0425af2e3440038 00000080:00080000:2.0:1767316247.056482:0:2489:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1767316247.056487:0:2489:0:(import.c:1204:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:2.0:1767316247.056490:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad93000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:2.0:1767316247.056496:0:2489:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0425af2e3440038 -> 0xa0425af2e3440038 00000080:00080000:2.0:1767316247.056498:0:2489:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1767316247.056506:0:2489:0:(recover.c:189:ptlrpc_wake_delayed()) @@@ waking (set 0000000053a4c325): req@ffff9d1f890e6d80 x1853163904538112/t0(0) o13->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316247.056995:0:45062:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1fc2173c00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316247.057480:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904537984:192.168.202.140@tcp:8:lfs.0 00010000:00080000:2.0:1767316247.059484:0:45062:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9d1f890e6d80 x1853163904538112/t0(0) o13->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:7/4 lens 224/368 e 0 to 0 dl 1767316263 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316247.059501:0:45062:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45062:1853163904538112:192.168.202.140@tcp:13:lfs.0 00020000:00000004:2.0:1767316247.059521:0:45062:0:(obd_class.h:1109:obd_statfs()) lustre-OST0001-osc-ffff9d1f98126000: update ffff9d1f8ad43520 cache blocks 3677/3683 objects 117728/118123 00000080:00200080:2.0:1767316247.059877:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.059893:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.059901:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.059905:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.059930:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.059935:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.059939:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.059942:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.059948:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.059951:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.059955:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.059958:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.059962:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.059965:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.059969:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.059972:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.059976:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.059979:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.059983:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.059986:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.059990:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.059994:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.059998:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060002:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060006:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060009:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060023:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060028:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060328:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060343:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060350:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060355:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060359:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060372:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060377:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060380:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060384:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060388:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060392:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060395:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060399:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060402:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060406:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060410:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060413:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060417:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060421:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060424:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060428:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060431:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060435:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060439:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060442:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060446:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060458:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060463:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060467:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060471:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060478:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060483:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060487:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060491:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060495:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060498:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060503:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060506:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060510:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060513:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060517:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060520:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060524:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060528:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060532:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060535:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060539:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060543:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060547:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060551:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060555:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060565:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060893:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060901:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060906:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060910:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060914:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060918:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060922:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060925:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060929:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060933:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060937:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060941:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060945:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060948:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060952:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060956:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060960:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060963:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060967:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060971:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060974:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060978:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060982:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.060986:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.060998:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.061002:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.061007:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.061010:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.061014:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.061018:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.061022:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.061025:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.061029:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.061110:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.061118:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.061122:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.061125:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.061129:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.061133:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.061137:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.061141:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.061144:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.061149:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.061152:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200080:2.0:1767316247.061156:0:45062:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7fffbadaa070 00020000:00000080:2.0:1767316247.061159:0:45062:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc0800 uarg=0000000000000000 00000080:00200008:2.0:1767316247.061175:0:45062:0:(file.c:424:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), flags 100000 00010000:00010000:3.0:1767316247.478870:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316248.505191:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767316249.526925:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1767316250.137392:0:45314:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 00010000:00010000:3.0:1767316250.552867:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767316251.584295:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1767316251.630804:0:45468:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00000100:00100000:0.0:1767316251.963006:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904538368:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316251.963117:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904538624:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316251.963150:0:2491:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1767316251.963157:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2491:1853163904538240:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316251.963181:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904538496:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316251.965863:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2491:1853163904538240:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316251.965884:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904538496:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316251.965896:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904538368:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316251.965906:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904538624:192.168.202.140@tcp:400:kworker.0 00010000:00010000:1.0:1767316252.604384:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1767316253.532322:0:45507:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1767316253.532335:0:45507:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:3.0:1767316253.532357:0:45507:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316253.532420:0:45507:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 45507 initiator: MDT0 00010000:00010000:3.0:1767316253.532431:0:45507:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 45507 initiator: MDT0 00010000:00010000:3.0:1767316253.532440:0:45507:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 45507 initiator: MDT0 00000100:00100000:3.0:1767316253.532465:0:45507:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45507:1853163904538752:192.168.202.140@tcp:101:lfs.0 00000100:00100000:3.0:1767316253.532570:0:45507:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f93ce8b40 going to sleep for 11 seconds 00000100:00100000:3.0:1767316253.556449:0:45507:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45507:1853163904538752:192.168.202.140@tcp:101:lfs.0 00010000:00010000:3.0:1767316253.556480:0:45507:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.556492:0:45507:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.556503:0:45507:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.556516:0:45507:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.556529:0:45507:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.556536:0:45507:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.556543:0:45507:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1767316253.556554:0:45507:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f92f73480 x1853163904538752/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316269 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1767316253.556568:0:45507:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.556582:0:45507:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1767316253.556589:0:45507:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1767316253.556835:0:45507:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:3.0:1767316253.556844:0:45507:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164910ca 00000080:00010000:3.0:1767316253.556850:0:45507:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:3.0:1767316253.556854:0:45507:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164910ca from it ffffb28141cafcb8 00010000:00010000:3.0:1767316253.556857:0:45507:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.556864:0:45507:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00000080:00200008:2.0:1767316253.560823:0:45507:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1767316253.560841:0:45507:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316253.560862:0:45507:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316253.560878:0:45507:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316253.560894:0:45507:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316253.560903:0:45507:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316253.560911:0:45507:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316253.560915:0:45507:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164910ca 00000080:00010000:2.0:1767316253.560922:0:45507:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164910ca from it ffffb28141cafcb8 00010000:00010000:2.0:1767316253.560926:0:45507:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316253.560935:0:45507:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316253.561074:0:45507:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316253.561083:0:45507:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316253.561090:0:45507:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316253.561098:0:45507:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316253.561105:0:45507:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316253.561109:0:45507:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316253.561111:0:45507:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164910ca 00000080:00010000:2.0:1767316253.561123:0:45507:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164910ca from it ffffb28141cafbb0 00010000:00010000:2.0:1767316253.561126:0:45507:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316253.561134:0:45507:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316253.562207:0:45507:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 44 00000080:00200008:2.0:1767316253.562218:0:45507:0:(file.c:1001:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), flags 100000 00000080:00200080:2.0:1767316253.562353:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.562368:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00800000:00000004:2.0:1767316253.562376:0:45507:0:(obd_class.h:1083:obd_statfs()) lustre-MDT0000-mdc-ffff9d1f98126000: age 1494, max_age 1500 00000100:00100000:2.0:1767316253.562421:0:45507:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45507:1853163904538880:192.168.202.140@tcp:41:lfs.0 00000100:00100000:2.0:1767316253.563136:0:45507:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f88daa180 going to sleep for 11 seconds 00000100:00100000:2.0:1767316253.570540:0:45507:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45507:1853163904538880:192.168.202.140@tcp:41:lfs.0 00800000:00000004:2.0:1767316253.570569:0:45507:0:(obd_class.h:1109:obd_statfs()) lustre-MDT0000-mdc-ffff9d1f98126000: update ffff9d1f90e03520 cache blocks 17229/17271 objects 499700/500067 00000080:00200080:2.0:1767316253.570708:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570725:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570733:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570737:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570741:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570745:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570749:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570753:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570757:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570760:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570779:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570783:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570787:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570791:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570795:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570798:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570803:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570807:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570811:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570815:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570819:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570822:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570826:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570830:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570834:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570837:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570841:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570844:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570848:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570851:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570856:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570859:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570863:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570866:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570870:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570880:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570897:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570902:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570906:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570910:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570915:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570918:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570922:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570926:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570930:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570933:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570937:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570941:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570946:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570949:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570953:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570956:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570960:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570963:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570966:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570970:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570973:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570977:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570981:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.570984:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.570997:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571002:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571006:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571010:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571013:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571017:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571020:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571023:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571027:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571030:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571033:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571036:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571040:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571044:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571048:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571052:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571056:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571059:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571063:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571067:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571071:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571075:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571079:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571082:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571086:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571097:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571103:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571106:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571110:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571114:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571118:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571121:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571125:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571128:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571131:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571134:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571138:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571141:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571145:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571149:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571154:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571157:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571160:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571163:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571167:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571170:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571173:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571177:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571180:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571184:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571196:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571200:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571204:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571208:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571211:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571214:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571218:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571222:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571226:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571229:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571233:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571236:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571240:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571244:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571248:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571251:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571256:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00800000:00000080:2.0:1767316253.571259:0:45507:0:(lmv_obd.c:905:lmv_iocontrol()) lustre-clilmv-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.571263:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.571355:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00020000:00000004:2.0:1767316253.571365:0:45507:0:(obd_class.h:1083:obd_statfs()) lustre-OST0000-osc-ffff9d1f98126000: age 1494, max_age 1500 00000100:00100000:2.0:1767316253.571406:0:45507:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45507:1853163904539008:192.168.202.140@tcp:13:lfs.0 00000100:00100000:2.0:1767316253.571476:0:45507:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f88daa180 going to sleep for 11 seconds 00000100:00100000:2.0:1767316253.574964:0:45507:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45507:1853163904539008:192.168.202.140@tcp:13:lfs.0 00020000:00000004:2.0:1767316253.574993:0:45507:0:(obd_class.h:1109:obd_statfs()) lustre-OST0000-osc-ffff9d1f98126000: update ffff9d1f8ad44df8 cache blocks 3675/3683 objects 117664/118623 00000080:00200080:2.0:1767316253.575062:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.575078:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00020000:00000004:2.0:1767316253.575085:0:45507:0:(obd_class.h:1083:obd_statfs()) lustre-OST0001-osc-ffff9d1f98126000: age 1494, max_age 1500 00000100:00100000:2.0:1767316253.575129:0:45507:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45507:1853163904539136:192.168.202.140@tcp:13:lfs.0 00000100:00100000:2.0:1767316253.575227:0:45507:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f88daa180 going to sleep for 11 seconds 00000100:00100000:2.0:1767316253.578851:0:45507:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45507:1853163904539136:192.168.202.140@tcp:13:lfs.0 00020000:00000004:2.0:1767316253.578878:0:45507:0:(obd_class.h:1109:obd_statfs()) lustre-OST0001-osc-ffff9d1f98126000: update ffff9d1f8ad43520 cache blocks 3677/3683 objects 117728/118123 00000080:00200080:2.0:1767316253.578938:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.578954:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.578960:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.578965:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.578969:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.578974:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.578978:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.578981:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.578986:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.578989:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.578993:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.578997:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579001:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579005:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579009:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579013:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579016:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579037:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579042:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579046:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579050:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579054:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579057:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579061:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579065:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579068:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579072:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579075:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579079:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579082:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579086:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579089:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579092:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579096:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579099:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579103:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579106:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579110:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579114:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579117:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579121:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579125:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579549:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579560:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579567:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579571:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579576:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579579:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579583:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579587:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579602:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579607:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579612:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579616:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579621:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579625:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579629:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579632:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579818:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579826:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579832:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579836:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579841:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579844:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579849:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579852:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579857:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579867:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579873:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579877:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579881:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579884:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579888:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579892:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579897:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579900:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579904:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579907:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579911:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579915:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579919:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579922:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579927:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579930:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579934:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579938:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579942:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579945:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579949:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579953:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579957:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579960:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579973:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579977:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579982:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579985:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579989:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.579993:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.579997:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580000:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.580004:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580008:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.580012:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580016:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.580019:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580022:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.580026:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580029:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.580032:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580035:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.580039:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580042:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.580045:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580049:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.580053:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580056:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.580060:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580071:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.580076:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580081:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.580085:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580088:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.580091:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580095:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200080:2.0:1767316253.580098:0:45507:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) cmd=c00866a4 arg=7ffc5b37ff90 00020000:00000080:2.0:1767316253.580102:0:45507:0:(lov_obd.c:958:lov_iocontrol()) lustre-clilov-ffff9d1f98126000: cmd=c00866a4 len=592 karg=ffff9d1f98bc3c00 uarg=0000000000000000 00000080:00200008:2.0:1767316253.580116:0:45507:0:(file.c:424:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), flags 100000 00010000:00010000:0.0:1767316253.626709:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767316253.650918:0:45526:0:(ldlm_resource.c:158:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff9d1f98126000 00010000:00010000:0.0:1767316253.650943:0:45526:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1767316253.651412:0:7208:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1767316253.651422:0:7208:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.651486:0:7208:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.651493:0:7208:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1767316253.651518:0:7208:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:45d767c3-7963-411a-be8f-bcbeb1c06075:7208:1853163904539264:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1767316253.651617:0:7208:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f93ce8b40 going to sleep for 11 seconds 00000100:00100000:1.0:1767316253.663394:0:7208:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:45d767c3-7963-411a-be8f-bcbeb1c06075:7208:1853163904539264:192.168.202.140@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1767316253.663427:0:7208:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910ca lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c96e0 expref: -99 pid: 45507 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316253.663466:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316253.663469:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316253.663472:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1767316253.724910:0:45532:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316253.724940:0:45532:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316253.725003:0:45532:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 45532 initiator: MDT0 00010000:00010000:1.0:1767316253.725013:0:45532:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 45532 initiator: MDT0 00010000:00010000:1.0:1767316253.725032:0:45532:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 45532 initiator: MDT0 00000100:00100000:1.0:1767316253.725057:0:45532:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3100 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45532:1853163904539392:192.168.202.140@tcp:101:lfs.0 00000100:00100000:1.0:1767316253.728359:0:45532:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8adfd000 going to sleep for 11 seconds 00000100:00100000:2.0:1767316253.738222:0:45532:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3100 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45532:1853163904539392:192.168.202.140@tcp:101:lfs.0 00010000:00010000:2.0:1767316253.738248:0:45532:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316253.738261:0:45532:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316253.738272:0:45532:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316253.738283:0:45532:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316253.738294:0:45532:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316253.738302:0:45532:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316253.738317:0:45532:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316253.738329:0:45532:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f918d3100 x1853163904539392/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316269 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316253.738343:0:45532:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316253.738357:0:45532:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1767316253.738363:0:45532:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316253.738390:0:45532:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1767316253.738396:0:45532:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164910d1 00000080:00010000:2.0:1767316253.738401:0:45532:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316253.738404:0:45532:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164910d1 from it ffffb281414fbbf8 00010000:00010000:2.0:1767316253.738407:0:45532:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316253.738415:0:45532:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316253.738440:0:45532:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316253.738456:0:45532:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=d202.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=27 00000080:00200000:2.0:1767316253.738461:0:45532:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=d202.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=0 00000002:00010000:2.0:1767316253.738481:0:45532:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d202.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1767316253.738522:0:45532:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 45532 initiator: MDT0 00010000:00010000:2.0:1767316253.738540:0:45532:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 45532 initiator: MDT0 00010000:00010000:2.0:1767316253.738550:0:45532:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 45532 initiator: MDT0 00000100:00100000:2.0:1767316253.738572:0:45532:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45532:1853163904539520:192.168.202.140@tcp:101:lfs.0 00000100:00100000:2.0:1767316253.738668:0:45532:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f88daa180 going to sleep for 11 seconds 00000100:00100000:3.0:1767316253.746548:0:45532:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45532:1853163904539520:192.168.202.140@tcp:101:lfs.0 00010000:00010000:3.0:1767316253.746573:0:45532:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.746605:0:45532:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.746613:0:45532:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.746642:0:45532:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.746653:0:45532:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.746665:0:45532:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.746672:0:45532:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.746711:0:45532:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1767316253.746722:0:45532:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=10 disposition=b, status=0 req@ffff9d1f890e6d80 x1853163904539520/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/584 e 0 to 0 dl 1767316269 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1767316253.746733:0:45532:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.746742:0:45532:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1127 (0 0) 00000080:00200000:3.0:1767316253.746748:0:45532:0:(namei.c:955:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1767316253.746770:0:45532:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x467:0x0], updating i_size 11776 00000080:00200000:3.0:1767316253.746773:0:45532:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x467:0x0](ffff9d1f884742c8): rc = 0 00000080:00010000:3.0:1767316253.746779:0:45532:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) for lock 0xa4df836b164910d8 00000080:00010000:3.0:1767316253.746790:0:45532:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) 00000080:00010000:3.0:1767316253.746793:0:45532:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164910d8 from it ffffb281414fbac8 00010000:00010000:3.0:1767316253.746802:0:45532:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.746810:0:45532:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316253.746842:0:45532:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), inode mode 41ed mask 1 00000080:00200000:3.0:1767316253.746868:0:45532:0:(namei.c:1354:ll_atomic_open()) VFS Op:name=f202.replay-single, dir=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), file ffff9d1f90912780, open_flags 18800, mode 0 opened 0 00000080:00200000:3.0:1767316253.746874:0:45532:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=f202.replay-single, dir=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), intent=open 00000002:00010000:3.0:1767316253.746895:0:45532:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f202.replay-single,[0x0:0x0:0x0]) in obj [0x200000bd1:0x467:0x0], intent: open flags 0304001 00010000:00010000:3.0:1767316253.746953:0:45532:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910df lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.746959:0:45532:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910df lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 45532 initiator: MDT0 00000020:00100000:3.0:1767316253.746970:0:45532:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1767316253.746977:0:45532:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f92f70000 x1853163904539776/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1767316253.746985:0:45532:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910df lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 45532 initiator: MDT0 00000100:00100000:3.0:1767316253.747008:0:45532:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45532:1853163904539776:192.168.202.140@tcp:101:lfs.0 00000100:00100000:3.0:1767316253.747177:0:45532:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f93ce8b40 going to sleep for 11 seconds 00000100:00100000:3.0:1767316253.753655:0:45532:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45532:1853163904539776:192.168.202.140@tcp:101:lfs.0 00010000:00010000:3.0:1767316253.753715:0:45532:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910df lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.753732:0:45532:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910df lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.753751:0:45532:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910df lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.753758:0:45532:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910df lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.753765:0:45532:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910df lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.753772:0:45532:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f8b384a00 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:3.0:1767316253.753776:0:45532:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910df lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.753782:0:45532:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910df lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.753788:0:45532:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910df lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.753795:0:45532:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1767316253.753801:0:45532:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910df lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.753807:0:45532:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384a00/0xa4df836b164910df lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 45532 initiator: MDT0 00000002:00100000:3.0:1767316253.753824:0:45532:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=b, status=-20 req@ffff9d1f92f70000 x1853163904539776/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 776/608 e 0 to 0 dl 1767316269 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1767316253.753838:0:45532:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000bd1:0x467:0x0] suppgids 0 -1: rc -20 00000080:00200000:3.0:1767316253.753984:0:45532:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:3.0:1767316253.754004:0:45532:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316253.754023:0:45532:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.754035:0:45532:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316253.754042:0:45532:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1767316253.754048:0:45532:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:3.0:1767316253.754052:0:45532:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164910d1 00000080:00010000:3.0:1767316253.754058:0:45532:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164910d1 from it ffffb281414fbbf8 00010000:00010000:3.0:1767316253.754061:0:45532:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316253.754070:0:45532:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316253.754085:0:45532:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:3.0:1767316253.754094:0:45532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d202.replay-single, flags=267 00000080:00200000:3.0:1767316253.754101:0:45532:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), inode mode 41ed mask 44 00000080:00200008:3.0:1767316253.754109:0:45532:0:(file.c:1001:ll_file_open()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), flags 304000 00000002:00010000:3.0:1767316253.754123:0:45532:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x467:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:3.0:1767316253.754168:0:45532:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b164910e6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.754175:0:45532:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b164910e6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 45532 initiator: MDT0 00000020:00100000:3.0:1767316253.754187:0:45532:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1767316253.754194:0:45532:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f92f70000 x1853163904540032/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1767316253.754226:0:45532:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b164910e6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 45532 initiator: MDT0 00000100:00100000:3.0:1767316253.754248:0:45532:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45532:1853163904540032:192.168.202.140@tcp:101:lfs.0 00000100:00100000:3.0:1767316253.754430:0:45532:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f93ce8b40 going to sleep for 11 seconds 00000100:00100000:3.0:1767316253.763102:0:45532:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45532:1853163904540032:192.168.202.140@tcp:101:lfs.0 00010000:00010000:3.0:1767316253.763130:0:45532:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b164910e6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.763144:0:45532:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b164910e6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.763151:0:45532:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b164910e6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.763158:0:45532:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b164910e6 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.763165:0:45532:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b164910e6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.763188:0:45532:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f8b384400 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:3.0:1767316253.763194:0:45532:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b164910e6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.763207:0:45532:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b164910e6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.763213:0:45532:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b164910e6 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.763220:0:45532:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1767316253.763240:0:45532:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b164910e6 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 45532 initiator: MDT0 00010000:00010000:3.0:1767316253.763247:0:45532:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384400/0xa4df836b164910e6 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 45532 initiator: MDT0 00000002:00100000:3.0:1767316253.763275:0:45532:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9d1f92f70000 x1853163904540032/t51539607553(51539607553) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 760/608 e 0 to 0 dl 1767316269 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1767316253.763289:0:45532:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f92f70000 x1853163904540032/t51539607553(51539607553) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 760/608 e 0 to 0 dl 1767316269 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1767316253.763301:0:45532:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1767316253.763322:0:45532:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x467:0x0], updating i_size 11776 00000002:00100000:3.0:1767316253.763337:0:45532:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f92f70000 x1853163904540032/t51539607553(51539607553) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 760/608 e 0 to 0 dl 1767316269 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:3.0:1767316253.763432:0:45532:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8) cmd=c0086915 arg=144e3d0 00000100:00100000:3.0:1767316253.763487:0:45532:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45532:1853163904540160:192.168.202.140@tcp:34:lfs.0 00000100:00100000:3.0:1767316253.763572:0:45532:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f93ce8b40 going to sleep for 11 seconds 00000100:00100000:3.0:1767316253.767048:0:45532:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45532:1853163904540160:192.168.202.140@tcp:34:lfs.0 00000080:00200008:3.0:1767316253.767349:0:45532:0:(file.c:424:ll_file_release()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), flags 304000 00000002:00100000:3.0:1767316253.767414:0:45532:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f92f70000 x1853163904540032/t51539607553(51539607553) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 760/608 e 0 to 0 dl 1767316269 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1767316253.767426:0:45532:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1767316253.767431:0:45532:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f92f71500 x1853163904540416/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1767316253.767454:0:45532:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45532:1853163904540416:192.168.202.140@tcp:35:lfs.0 00000100:00100000:3.0:1767316253.767533:0:45532:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f93ce8b40 going to sleep for 17 seconds 00000100:00100000:3.0:1767316253.770538:0:45532:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job lfs:45d767c3-7963-411a-be8f-bcbeb1c06075:45532:1853163904540416:192.168.202.140@tcp:35:lfs.0 00000002:00100000:3.0:1767316253.770572:0:45532:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f92f70000 x1853163904540032/t51539607553(51539607553) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 760/608 e 0 to 0 dl 1767316269 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1767316254.651634:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316255.681301:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316256.694862:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1767316257.079106:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904540544:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316257.079210:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904540800:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316257.079245:0:2490:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1767316257.079251:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904540672:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316257.079273:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904540928:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316257.082851:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904540544:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316257.087651:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904540672:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316257.087672:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904540928:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316257.087725:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904540800:192.168.202.140@tcp:400:kworker.0 00010000:00010000:3.0:1767316257.719363:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767316258.751514:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316259.768476:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1767316260.450152:0:46096:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 203: resend can hit original request ========================================================== 20:10:58 (1767316258) 00010000:00010000:0.0:1767316260.792104:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:01200004:0.0:1767316260.906263:0:46117:0:(super25.c:139:lustre_fill_super()) VFS Op: sb ffff9d1f90a81800 00000020:01000004:0.0:1767316260.906291:0:46117:0:(obd_mount.c:901:lmd_print()) mount data: 00000020:01000004:0.0:1767316260.906293:0:46117:0:(obd_mount.c:903:lmd_print()) profile: lustre-client 00000020:01000004:0.0:1767316260.906295:0:46117:0:(obd_mount.c:904:lmd_print()) device: 192.168.202.140@tcp:/lustre 00000020:01000004:0.0:1767316260.906296:0:46117:0:(obd_mount.c:907:lmd_print()) options: user_xattr,flock 00000080:01000004:0.0:1767316260.906298:0:46117:0:(super25.c:182:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:0.0:1767316260.906316:0:46117:0:(obd_mount.c:392:lustre_start_mgc()) MGC192.168.202.140@tcp:Set MGC reconnect 1 10000000:01000000:0.0:1767316260.906320:0:46117:0:(mgc_request.c:1068:mgc_set_info_async()) InitRecov MGC192.168.202.140@tcp 1/d0:i0:r0:or0:FULL 00000080:00200000:0.0:1767316260.906325:0:46117:0:(llite_lib.c:1391:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff9d1f90a81800 (sb ffff9d1f90a81800) 00000080:01000000:0.0:1767316260.906402:0:46117:0:(llite_lib.c:1113:ll_options()) Parsing opts user_xattr,flock 00000080:00000004:0.0:1767316260.906404:0:46117:0:(llite_lib.c:1123:ll_options()) next opt=user_xattr 00000080:00000004:0.0:1767316260.906407:0:46117:0:(llite_lib.c:1123:ll_options()) next opt=flock 00000080:01000000:0.0:1767316260.906414:0:46117:0:(llite_lib.c:1425:ll_fill_super()) llite sb uuid: 5b9cdf94-0180-42f8-9a71-beb807b227b6 10000000:01000000:0.0:1767316260.909852:0:46117:0:(mgc_request.c:1952:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:0.0:1767316260.909859:0:46117:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff9d1f90a81800 10000000:01000000:0.0:1767316260.909867:0:46117:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff9d1f90a81800 10000000:01000000:0.0:1767316260.909872:0:46117:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:0.0:1767316260.909876:0:46117:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff9d1f90a81800 10000000:01000000:0.0:1767316260.909880:0:46117:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:0.0:1767316260.909884:0:46117:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff9d1f90a81800 10000000:01000000:0.0:1767316260.909887:0:46117:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:0.0:1767316260.909891:0:46117:0:(mgc_request.c:1812:mgc_process_log()) Process log lustre-client-ffff9d1f90a81800 from 1 10000000:01000000:0.0:1767316260.909894:0:46117:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:0.0:1767316260.909946:0:46117:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316260.909958:0:46117:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316260.909971:0:46117:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1767316260.910002:0:46117:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904541056:192.168.202.140@tcp:101:mount.lustre.0 00000100:00100000:0.0:1767316260.912234:0:46117:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8af11600 going to sleep for 11 seconds 00000100:00100000:3.0:1767316260.922608:0:46117:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904541056:192.168.202.140@tcp:101:mount.lustre.0 00010000:00010000:3.0:1767316260.922638:0:46117:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316260.922652:0:46117:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000000000000 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00000100:00100000:3.0:1767316260.922825:0:46117:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904541184:192.168.202.140@tcp:501:mount.lustre.0 00000100:00100000:3.0:1767316260.922901:0:46117:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b540 going to sleep for 11 seconds 00000100:00100000:0.0:1767316260.936038:0:46117:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904541184:192.168.202.140@tcp:501:mount.lustre.0 00000100:00100000:0.0:1767316260.936115:0:46117:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904541312:192.168.202.140@tcp:503:mount.lustre.0 00000100:00100000:0.0:1767316260.936207:0:46117:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8af11180 going to sleep for 11 seconds 00000100:00100000:0.0:1767316260.947206:0:46117:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904541312:192.168.202.140@tcp:503:mount.lustre.0 00000100:00100000:2.0:1767316260.947616:0:46121:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46121:1853163904541440:192.168.202.140@tcp:502:llog_process_th.0 00000100:00100000:2.0:1767316260.947904:0:46121:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f98865900 going to sleep for 11 seconds 00000100:00100000:2.0:1767316260.953430:0:46121:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46121:1853163904541440:192.168.202.140@tcp:502:llog_process_th.0 00000020:01000000:2.0:1767316260.953580:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1767316260.953598:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.953601:0:46121:0:(obd_config.c:1477:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:2.0:1767316260.953609:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.953612:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1767316260.953653:0:46121:0:(genops.c:412:class_newdev()) Allocate new device lustre-clilov-ffff9d1f90a81800 (ffff9d1f8b3998d8) 00000020:00000080:2.0:1767316260.953904:0:46121:0:(obd_config.c:657:class_attach()) OBD: dev 6 attached type lov with refcount 2 00000020:01000000:2.0:1767316260.953913:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.953915:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1767316260.954347:0:46121:0:(obd_config.c:761:class_setup()) finished setup of obd lustre-clilov-ffff9d1f90a81800 (uuid 5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000020:01000000:2.0:1767316260.954354:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1767316260.954358:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.954359:0:46121:0:(obd_config.c:1477:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:2.0:1767316260.954363:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1767316260.954366:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.954367:0:46121:0:(obd_config.c:1477:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:2.0:1767316260.954371:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.954382:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1767316260.954391:0:46121:0:(genops.c:412:class_newdev()) Allocate new device lustre-clilmv-ffff9d1f90a81800 (ffff9d1f8b39e360) 00000020:00000080:2.0:1767316260.954403:0:46121:0:(obd_config.c:657:class_attach()) OBD: dev 7 attached type lmv with refcount 2 00000020:01000000:2.0:1767316260.954409:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.954412:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1767316260.956026:0:46121:0:(obd_config.c:761:class_setup()) finished setup of obd lustre-clilmv-ffff9d1f90a81800 (uuid 5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000020:01000000:2.0:1767316260.956038:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1767316260.956045:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.956047:0:46121:0:(obd_config.c:1477:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:2.0:1767316260.956051:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1767316260.956056:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.956057:0:46121:0:(obd_config.c:1477:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:2.0:1767316260.956062:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf007 00000020:00000080:2.0:1767316260.956064:0:46121:0:(obd_config.c:1430:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:2.0:1767316260.956068:0:46121:0:(obd_config.c:1074:class_add_profile()) Add profile lustre-client 00000020:01000000:2.0:1767316260.956075:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1767316260.956078:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.956080:0:46121:0:(obd_config.c:1477:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:2.0:1767316260.956082:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1767316260.956087:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.956088:0:46121:0:(obd_config.c:1477:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:2.0:1767316260.956091:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1767316260.956098:0:46121:0:(obd_config.c:1401:class_process_config()) adding mapping from uuid 192.168.202.140@tcp to nid 0x20000c0a8ca8c (192.168.202.140@tcp) 00000020:01000000:2.0:1767316260.956881:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.956891:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1767316260.956907:0:46121:0:(genops.c:412:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff9d1f90a81800 (ffff9d1f8b398000) 00000020:00000080:2.0:1767316260.956928:0:46121:0:(obd_config.c:657:class_attach()) OBD: dev 8 attached type mdc with refcount 2 00000020:01000000:2.0:1767316260.956935:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.956938:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1767316260.957055:0:46121:0:(ldlm_lib.c:102:import_set_conn()) imp ffff9d1f8ad2f000@lustre-MDT0000-mdc-ffff9d1f90a81800: add connection 192.168.202.140@tcp at head 00010000:00010000:2.0:1767316260.965149:0:46121:0:(ldlm_pool.c:924:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff9d1f90a81800-4 is initialized 00000040:01000000:2.0:1767316260.965402:0:46121:0:(llog_obd.c:195:llog_setup()) obd lustre-MDT0000-mdc-ffff9d1f90a81800 ctxt 13 is initialized 00000020:00000080:2.0:1767316260.965481:0:46121:0:(obd_config.c:761:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff9d1f90a81800 (uuid 5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000020:01000000:2.0:1767316260.965502:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.965506:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1767316260.965555:0:46121:0:(lmv_obd.c:425:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:2.0:1767316260.965604:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1767316260.965607:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.965609:0:46121:0:(obd_config.c:1477:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:2.0:1767316260.965611:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1767316260.965614:0:46121:0:(obd_config.c:1797:class_config_llog_handler()) SKIP #8 00000020:01000000:2.0:1767316260.965616:0:46121:0:(obd_config.c:1823:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1767316260.965618:0:46121:0:(obd_config.c:1823:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1767316260.965619:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1767316260.965622:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.965623:0:46121:0:(obd_config.c:1477:class_process_config()) marker 8 (0x6) lustre sys.timeout 00000020:01000000:2.0:1767316260.965626:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1767316260.965632:0:46121:0:(obd_mount.c:939:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.140@tcp:/lustre 00000020:00000080:2.0:1767316260.965635:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.965637:0:46121:0:(obd_config.c:1477:class_process_config()) marker 13 (0x1) lustre-OST0001 add osc 00000020:00000080:2.0:1767316260.965639:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1767316260.965645:0:46121:0:(obd_config.c:1401:class_process_config()) adding mapping from uuid 192.168.202.140@tcp to nid 0x20000c0a8ca8c (192.168.202.140@tcp) 00000020:01000000:2.0:1767316260.965709:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.965712:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1767316260.965729:0:46121:0:(genops.c:412:class_newdev()) Allocate new device lustre-OST0001-osc-ffff9d1f90a81800 (ffff9d1f8b39b1b0) 00000020:00000080:2.0:1767316260.965746:0:46121:0:(obd_config.c:657:class_attach()) OBD: dev 9 attached type osc with refcount 2 00000020:01000000:2.0:1767316260.965752:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.965755:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1767316260.965838:0:46121:0:(ldlm_lib.c:102:import_set_conn()) imp ffff9d1f942fd800@lustre-OST0001-osc-ffff9d1f90a81800: add connection 192.168.202.140@tcp at head 00010000:00010000:2.0:1767316260.966029:0:46121:0:(ldlm_pool.c:924:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff9d1f90a81800-5 is initialized 00000020:00000080:2.0:1767316260.966227:0:46121:0:(obd_config.c:761:class_setup()) finished setup of obd lustre-OST0001-osc-ffff9d1f90a81800 (uuid 5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000020:01000000:2.0:1767316260.966234:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.966236:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1767316260.966253:0:46121:0:(lov_obd.c:469:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:2.0:1767316260.966260:0:46121:0:(lov_obd.c:498:lov_add_target()) lustre-clilov-ffff9d1f90a81800: Adding target lustre-OST0001_UUID 00020000:01000000:2.0:1767316260.966267:0:46121:0:(lov_obd.c:520:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=1 00000020:01000000:2.0:1767316260.966276:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1767316260.966280:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.966281:0:46121:0:(obd_config.c:1477:class_process_config()) marker 13 (0x2) lustre-OST0001 add osc 00000020:01000000:2.0:1767316260.966284:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1767316260.966286:0:46121:0:(obd_config.c:1797:class_config_llog_handler()) SKIP #14 00000020:01000000:2.0:1767316260.966287:0:46121:0:(obd_config.c:1823:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1767316260.966289:0:46121:0:(obd_config.c:1823:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1767316260.966291:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1767316260.966293:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.966294:0:46121:0:(obd_config.c:1477:class_process_config()) marker 14 (0x6) lustre sys.timeout 00000020:01000000:2.0:1767316260.966296:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1767316260.966299:0:46121:0:(obd_mount.c:939:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.140@tcp:/lustre 00000020:00000080:2.0:1767316260.966302:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.966303:0:46121:0:(obd_config.c:1477:class_process_config()) marker 19 (0x1) lustre-OST0000 add osc 00000020:00000080:2.0:1767316260.966305:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1767316260.966308:0:46121:0:(obd_config.c:1401:class_process_config()) adding mapping from uuid 192.168.202.140@tcp to nid 0x20000c0a8ca8c (192.168.202.140@tcp) 00000020:01000000:2.0:1767316260.966320:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.966323:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1767316260.966328:0:46121:0:(genops.c:412:class_newdev()) Allocate new device lustre-OST0000-osc-ffff9d1f90a81800 (ffff9d1f8b39ca88) 00000020:00000080:2.0:1767316260.966337:0:46121:0:(obd_config.c:657:class_attach()) OBD: dev 10 attached type osc with refcount 2 00000020:01000000:2.0:1767316260.966341:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.966343:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1767316260.966386:0:46121:0:(ldlm_lib.c:102:import_set_conn()) imp ffff9d1f942fe800@lustre-OST0000-osc-ffff9d1f90a81800: add connection 192.168.202.140@tcp at head 00010000:00010000:2.0:1767316260.972932:0:46121:0:(ldlm_pool.c:924:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff9d1f90a81800-6 is initialized 00000020:00000080:2.0:1767316260.975347:0:46121:0:(obd_config.c:761:class_setup()) finished setup of obd lustre-OST0000-osc-ffff9d1f90a81800 (uuid 5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000020:01000000:2.0:1767316260.975365:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.975368:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1767316260.975391:0:46121:0:(lov_obd.c:469:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:2.0:1767316260.975401:0:46121:0:(lov_obd.c:498:lov_add_target()) lustre-clilov-ffff9d1f90a81800: Adding target lustre-OST0000_UUID 00020000:01000000:2.0:1767316260.975411:0:46121:0:(lov_obd.c:520:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=2 00000020:01000000:2.0:1767316260.975419:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1767316260.975422:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.975423:0:46121:0:(obd_config.c:1477:class_process_config()) marker 19 (0x2) lustre-OST0000 add osc 00000020:01000000:2.0:1767316260.975427:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1767316260.975429:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.975430:0:46121:0:(obd_config.c:1477:class_process_config()) marker 20 (0x1) lustre sys.timeout 00000020:00000080:2.0:1767316260.975432:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf009 00000020:00000080:2.0:1767316260.975433:0:46121:0:(obd_config.c:1451:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:2.0:1767316260.975435:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1767316260.975436:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.975437:0:46121:0:(obd_config.c:1477:class_process_config()) marker 20 (0x2) lustre sys.timeout 00000020:01000000:2.0:1767316260.975438:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1767316260.975439:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.975440:0:46121:0:(obd_config.c:1477:class_process_config()) marker 24 (0x1) lustre sys.jobid_var 00000020:00000080:2.0:1767316260.975441:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf00f 00000020:01000000:2.0:1767316260.975445:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1767316260.975446:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.975447:0:46121:0:(obd_config.c:1477:class_process_config()) marker 24 (0x2) lustre sys.jobid_var 00000020:01000000:2.0:1767316260.975448:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1767316260.975449:0:46121:0:(obd_config.c:1797:class_config_llog_handler()) SKIP #25 00000020:01000000:2.0:1767316260.975450:0:46121:0:(obd_config.c:1823:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1767316260.975451:0:46121:0:(obd_config.c:1823:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1767316260.975452:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1767316260.975453:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.975454:0:46121:0:(obd_config.c:1477:class_process_config()) marker 25 (0x6) lustre-clilov new lustre.pool_134 00000020:01000000:2.0:1767316260.975455:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1767316260.975456:0:46121:0:(obd_config.c:1797:class_config_llog_handler()) SKIP #27 00000020:01000000:2.0:1767316260.975457:0:46121:0:(obd_config.c:1823:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1767316260.975458:0:46121:0:(obd_config.c:1823:class_config_llog_handler()) skipping 0x6 00000100:00100000:2.0:1767316260.975662:0:46121:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46121:1853163904541568:192.168.202.140@tcp:502:llog_process_th.0 00000100:00100000:2.0:1767316260.975880:0:46121:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f98865900 going to sleep for 11 seconds 00000100:00100000:2.0:1767316260.991404:0:46121:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46121:1853163904541568:192.168.202.140@tcp:502:llog_process_th.0 00000020:01000000:2.0:1767316260.991536:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1767316260.991544:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.991547:0:46121:0:(obd_config.c:1477:class_process_config()) marker 27 (0x6) lustre-clilov add lustre.pool_134.lustre-OST0001_UUID 00000020:01000000:2.0:1767316260.991554:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1767316260.991559:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.991560:0:46121:0:(obd_config.c:1477:class_process_config()) marker 28 (0x1) lustre-clilov rem lustre.pool_134.lustre-OST0001_UUID 00000020:01000000:2.0:1767316260.991566:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd ce022, instance name: lustre-clilov-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.991569:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: ce022 00000020:01000000:2.0:1767316260.991580:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1767316260.991582:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.991583:0:46121:0:(obd_config.c:1477:class_process_config()) marker 28 (0x2) lustre-clilov rem lustre.pool_134.lustre-OST0001_UUID 00000020:01000000:2.0:1767316260.991584:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1767316260.991586:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.991586:0:46121:0:(obd_config.c:1477:class_process_config()) marker 30 (0x1) lustre-clilov del lustre.pool_134 00000020:01000000:2.0:1767316260.991588:0:46121:0:(obd_config.c:1903:class_config_llog_handler()) cmd ce023, instance name: lustre-clilov-ffff9d1f90a81800 00000020:00000080:2.0:1767316260.991590:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: ce023 00000100:00100000:2.0:1767316260.991624:0:46121:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46121:1853163904541696:192.168.202.140@tcp:502:llog_process_th.0 00000100:00100000:2.0:1767316260.991718:0:46121:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f966f80c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316260.998301:0:46121:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46121:1853163904541696:192.168.202.140@tcp:502:llog_process_th.0 00000020:01000000:2.0:1767316260.998338:0:46121:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1767316260.998344:0:46121:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1767316260.998345:0:46121:0:(obd_config.c:1477:class_process_config()) marker 30 (0x2) lustre-clilov del lustre.pool_134 00000040:00080000:2.0:1767316260.998350:0:46121:0:(llog.c:854:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 55 count 54 00000020:01000000:0.0:1767316260.998432:0:46117:0:(obd_config.c:2043:class_config_parse_llog()) Processed log lustre-client gen 1-54 (rc=0) 10000000:01000000:0.0:1767316260.998455:0:46117:0:(mgc_request.c:1889:mgc_process_log()) MGC192.168.202.140@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:0.0:1767316260.998478:0:46117:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316260.998489:0:46117:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 10000000:01000000:0.0:1767316260.998497:0:46117:0:(mgc_request.c:1812:mgc_process_log()) Process log lustre-cliir-ffff9d1f90a81800 from 1 10000000:01000000:0.0:1767316260.998500:0:46117:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1767316260.998549:0:46117:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316260.998558:0:46117:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316260.998581:0:46117:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1767316260.998602:0:46117:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904541824:192.168.202.140@tcp:101:mount.lustre.0 00000100:00100000:0.0:1767316260.998789:0:46117:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f98738f00 going to sleep for 11 seconds 00000100:00100000:0.0:1767316261.001085:0:46117:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904541824:192.168.202.140@tcp:101:mount.lustre.0 00010000:00010000:0.0:1767316261.001119:0:46117:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316261.001132:0:46117:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1767316261.001967:0:46117:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904541952:192.168.202.140@tcp:256:mount.lustre.0 00000100:00100000:0.0:1767316261.002107:0:46117:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f98738f00 going to sleep for 11 seconds 00000100:00100000:0.0:1767316261.007933:0:46117:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904541952:192.168.202.140@tcp:256:mount.lustre.0 00000020:00000080:0.0:1767316261.007988:0:46117:0:(obd_config.c:1389:class_process_config()) processing cmd: cf00f 10000000:01000000:0.0:1767316261.008829:0:46117:0:(mgc_request.c:1889:mgc_process_log()) MGC192.168.202.140@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:0.0:1767316261.008842:0:46117:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316261.008853:0:46117:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 10000000:01000000:0.0:1767316261.008860:0:46117:0:(mgc_request.c:1812:mgc_process_log()) Process log params-ffff9d1f90a81800 from 1 10000000:01000000:0.0:1767316261.008862:0:46117:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:0.0:1767316261.008895:0:46117:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316261.008913:0:46117:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316261.008921:0:46117:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1767316261.008938:0:46117:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904542080:192.168.202.140@tcp:101:mount.lustre.0 00000100:00100000:0.0:1767316261.009011:0:46117:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f98738f00 going to sleep for 11 seconds 00000100:00100000:3.0:1767316261.013653:0:46117:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904542080:192.168.202.140@tcp:101:mount.lustre.0 00010000:00010000:3.0:1767316261.013900:0:46117:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316261.013916:0:46117:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1767316261.014069:0:46117:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904542208:192.168.202.140@tcp:501:mount.lustre.0 00000100:00100000:2.0:1767316261.014159:0:46117:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f966f80c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316261.017294:0:46117:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904542208:192.168.202.140@tcp:501:mount.lustre.0 00000100:00100000:2.0:1767316261.017374:0:46117:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904542336:192.168.202.140@tcp:503:mount.lustre.0 00000100:00100000:2.0:1767316261.017479:0:46117:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f966f80c0 going to sleep for 11 seconds 00000100:00100000:1.0:1767316261.031871:0:46117:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46117:1853163904542336:192.168.202.140@tcp:503:mount.lustre.0 00000100:00100000:0.0:1767316261.038961:0:46128:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46128:1853163904542464:192.168.202.140@tcp:502:llog_process_th.0 00000100:00100000:0.0:1767316261.040149:0:46128:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f88daaf00 going to sleep for 11 seconds 00000100:00100000:3.0:1767316261.054077:0:46128:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46128:1853163904542464:192.168.202.140@tcp:502:llog_process_th.0 00000020:01000000:3.0:1767316261.054114:0:46128:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1767316261.054122:0:46128:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1767316261.054141:0:46128:0:(obd_config.c:1477:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:3.0:1767316261.054149:0:46128:0:(obd_config.c:1903:class_config_llog_handler()) cmd ce032, instance name: general-ffff9d1f90a81800 00000020:00000080:3.0:1767316261.054151:0:46128:0:(obd_config.c:1389:class_process_config()) processing cmd: ce032 00000020:01000000:3.0:1767316261.054159:0:46128:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1767316261.054162:0:46128:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1767316261.054163:0:46128:0:(obd_config.c:1477:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:3.0:1767316261.054165:0:46128:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1767316261.054168:0:46128:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1767316261.054169:0:46128:0:(obd_config.c:1477:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:3.0:1767316261.054172:0:46128:0:(obd_config.c:1903:class_config_llog_handler()) cmd ce032, instance name: general-ffff9d1f90a81800 00000020:00000080:3.0:1767316261.054174:0:46128:0:(obd_config.c:1389:class_process_config()) processing cmd: ce032 00000100:00100000:3.0:1767316261.054258:0:46128:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46128:1853163904542592:192.168.202.140@tcp:502:llog_process_th.0 00000100:00100000:3.0:1767316261.054334:0:46128:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b600 going to sleep for 11 seconds 00000100:00100000:3.0:1767316261.074888:0:46128:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:46128:1853163904542592:192.168.202.140@tcp:502:llog_process_th.0 00000020:01000000:0.0:1767316261.075356:0:46128:0:(obd_config.c:1788:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1767316261.075368:0:46128:0:(obd_config.c:1389:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1767316261.075370:0:46128:0:(obd_config.c:1477:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000040:00080000:0.0:1767316261.075377:0:46128:0:(llog.c:854:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 7 count 7 00000020:01000000:1.0:1767316261.075793:0:46117:0:(obd_config.c:2043:class_config_parse_llog()) Processed log params gen 1-6 (rc=0) 10000000:01000000:1.0:1767316261.075813:0:46117:0:(mgc_request.c:1889:mgc_process_log()) MGC192.168.202.140@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:1.0:1767316261.075831:0:46117:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316261.075843:0:46117:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00000080:01000000:1.0:1767316261.075860:0:46117:0:(llite_lib.c:1495:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:1.0:1767316261.075883:0:46117:0:(genops.c:1402:class_connect()) connect: client 5b9cdf94-0180-42f8-9a71-beb807b227b6, cookie 0xa4df836b16491125 00800000:01000000:1.0:1767316261.075904:0:46117:0:(lmv_obd.c:498:lmv_check_connect()) Time to connect 5b9cdf94-0180-42f8-9a71-beb807b227b6 to lustre-clilmv-ffff9d1f90a81800 00800000:01000000:1.0:1767316261.075911:0:46117:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff9d1f90a81800(5b9cdf94-0180-42f8-9a71-beb807b227b6) - lustre-MDT0000_UUID, 5b9cdf94-0180-42f8-9a71-beb807b227b6 00000020:00000080:1.0:1767316261.075925:0:46117:0:(genops.c:1402:class_connect()) connect: client 5b9cdf94-0180-42f8-9a71-beb807b227b6, cookie 0xa4df836b1649112c 00000100:00080000:1.0:1767316261.075933:0:46117:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f90a81800: connect to NID 192.168.202.140@tcp last attempt 0 00000100:00080000:1.0:1767316261.075945:0:46117:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f90a81800: import ffff9d1f8ad2f000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767316261.075958:0:46117:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1767316261.075984:0:46117:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3800 x1853163904542720/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316261.076081:0:46117:0:(pinger.c:371:ptlrpc_pinger_add_import()) adding pingable import 5b9cdf94-0180-42f8-9a71-beb807b227b6->lustre-MDT0000_UUID 00000100:00100000:0.0:1767316261.076814:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5b9cdf94-0180-42f8-9a71-beb807b227b6:2489:1853163904542720:192.168.202.140@tcp:38:mount.lustre.0 00800000:01000000:1.0:1767316261.077412:0:46117:0:(lmv_obd.c:395:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff9d1f90a81800(5b9cdf94-0180-42f8-9a71-beb807b227b6) successfully (5) 00800000:00000004:1.0:1767316261.077417:0:46117:0:(obd_class.h:1049:obd_statfs_async()) lustre-MDT0000-mdc-ffff9d1f90a81800: age 508, max_age 0 00000100:00080000:1.0:1767316261.077462:0:2490:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9d1f918d0700 x1853163904542848/t0(0) o41->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316261.077519:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904543104:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316261.077564:0:2490:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1767316261.077570:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904542976:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316261.077587:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904543232:192.168.202.140@tcp:400:kworker.0 00000080:00000004:1.0:1767316261.077621:0:46117:0:(obd_class.h:1083:obd_statfs()) lustre-clilmv-ffff9d1f90a81800: age 508, max_age 1507 00800000:00000004:1.0:1767316261.077628:0:46117:0:(obd_class.h:1083:obd_statfs()) lustre-MDT0000-mdc-ffff9d1f90a81800: age 508, max_age 1507 00000100:00080000:1.0:1767316261.077647:0:46117:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9d1f918d1c00 x1853163904543360/t0(0) o41->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316261.077655:0:46117:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a189cc0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316261.088973:0:2489:0:(client.c:3081:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d1f90a81800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1767316261.088991:0:2489:0:(import.c:1123:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff9d1f90a81800: connect to target with instance 12 00000100:00080000:2.0:1767316261.089001:0:2489:0:(import.c:951:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff9d1f90a81800: Resetting ns_connect_flags to server flags: 0xae7a5e7be344d3b8 00000100:00080000:2.0:1767316261.089023:0:2489:0:(import.c:1204:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:2.0:1767316261.089027:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad2f000 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:3.0:1767316261.090035:0:2489:0:(recover.c:189:ptlrpc_wake_delayed()) @@@ waking (set 00000000ac11c007): req@ffff9d1f918d0700 x1853163904542848/t0(0) o41->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767316261.090056:0:2489:0:(recover.c:189:ptlrpc_wake_delayed()) @@@ waking (set 00000000351c8faf): req@ffff9d1f918d1c00 x1853163904543360/t0(0) o41->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316261.090072:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5b9cdf94-0180-42f8-9a71-beb807b227b6:2489:1853163904542720:192.168.202.140@tcp:38:mount.lustre.0 00000100:00100000:3.0:1767316261.090415:0:46117:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f8a189cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1767316261.092777:0:2490:0:(client.c:3081:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d1f98126000: committing for last_committed 51539607554 gen 2 00000100:00100000:0.0:1767316261.092793:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904542976:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316261.092806:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904543232:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316261.092815:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904543104:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316261.095168:0:46117:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:5b9cdf94-0180-42f8-9a71-beb807b227b6:46117:1853163904543360:192.168.202.140@tcp:41:mount.lustre.0 00800000:00000004:3.0:1767316261.095200:0:46117:0:(obd_class.h:1109:obd_statfs()) lustre-MDT0000-mdc-ffff9d1f90a81800: update ffff9d1f8b398370 cache blocks 17229/17271 objects 499700/500067 00000080:00000004:3.0:1767316261.095208:0:46117:0:(obd_class.h:1109:obd_statfs()) lustre-clilmv-ffff9d1f90a81800: update ffff9d1f8b39e6d0 cache blocks 17229/17271 objects 499700/500067 00000100:00100000:3.0:1767316261.095357:0:46117:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:5b9cdf94-0180-42f8-9a71-beb807b227b6:46117:1853163904543488:192.168.202.140@tcp:40:mount.lustre.0 00000100:00100000:3.0:1767316261.095436:0:46117:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b600 going to sleep for 11 seconds 00000100:00100000:0.0:1767316261.097658:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:5b9cdf94-0180-42f8-9a71-beb807b227b6:0:1853163904542848:192.168.202.140@tcp:41:mount.lustre.0 00000100:00100000:3.0:1767316261.098107:0:46117:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:5b9cdf94-0180-42f8-9a71-beb807b227b6:46117:1853163904543488:192.168.202.140@tcp:40:mount.lustre.0 00000080:00000004:3.0:1767316261.098127:0:46117:0:(llite_lib.c:511:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:3.0:1767316261.098141:0:46117:0:(llite_lib.c:674:client_common_fill_super()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34668544 ocd_grant: 3407872 00020000:01000000:3.0:1767316261.098145:0:46117:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:3.0:1767316261.098157:0:46117:0:(genops.c:1402:class_connect()) connect: client 5b9cdf94-0180-42f8-9a71-beb807b227b6, cookie 0xa4df836b16491133 00000020:00000080:3.0:1767316261.098177:0:46117:0:(genops.c:1402:class_connect()) connect: client LOV_OSC_UUID, cookie 0xa4df836b1649113a 00000100:00080000:3.0:1767316261.098186:0:46117:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d1f90a81800: connect to NID 192.168.202.140@tcp last attempt 0 00000100:00080000:3.0:1767316261.098195:0:46117:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d1f90a81800: import ffff9d1f942fe800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767316261.098214:0:46117:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34668544 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1767316261.098237:0:46117:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70000 x1853163904543616/t0(0) o8->lustre-OST0000-osc-ffff9d1f90a81800@192.168.202.140@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767316261.098274:0:46117:0:(pinger.c:371:ptlrpc_pinger_add_import()) adding pingable import 5b9cdf94-0180-42f8-9a71-beb807b227b6->lustre-OST0000_UUID 00020000:01000000:3.0:1767316261.098293:0:46117:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff9d1f90a81800) active 00000080:00000004:3.0:1767316261.098313:0:46117:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0x0 -> 0xa0465af2e3640038 00000020:00000080:3.0:1767316261.098325:0:46117:0:(genops.c:1402:class_connect()) connect: client LOV_OSC_UUID, cookie 0xa4df836b16491141 00000100:00080000:3.0:1767316261.098329:0:46117:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d1f90a81800: connect to NID 192.168.202.140@tcp last attempt 0 00000100:00080000:3.0:1767316261.098334:0:46117:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d1f90a81800: import ffff9d1f942fd800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767316261.098339:0:46117:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34668544 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1767316261.098349:0:46117:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73b80 x1853163904543744/t0(0) o8->lustre-OST0001-osc-ffff9d1f90a81800@192.168.202.140@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767316261.098358:0:46117:0:(pinger.c:371:ptlrpc_pinger_add_import()) adding pingable import 5b9cdf94-0180-42f8-9a71-beb807b227b6->lustre-OST0001_UUID 00020000:01000000:3.0:1767316261.098363:0:46117:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff9d1f90a81800) active 00000080:00000004:3.0:1767316261.098370:0:46117:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0x0 -> 0xa0465af2e3640038 00000100:00100000:3.0:1767316261.098407:0:46117:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:5b9cdf94-0180-42f8-9a71-beb807b227b6:46117:1853163904543872:192.168.202.140@tcp:33:mount.lustre.0 00000100:00100000:3.0:1767316261.098541:0:46117:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b600 going to sleep for 11 seconds 00000100:00100000:2.0:1767316261.098855:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5b9cdf94-0180-42f8-9a71-beb807b227b6:2489:1853163904543616:192.168.202.140@tcp:8:mount.lustre.0 00000100:00100000:2.0:1767316261.098913:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5b9cdf94-0180-42f8-9a71-beb807b227b6:2489:1853163904543744:192.168.202.140@tcp:8:mount.lustre.0 00000100:00100000:2.0:1767316261.107424:0:2489:0:(client.c:3081:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9d1f90a81800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1767316261.107440:0:2489:0:(import.c:1123:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff9d1f90a81800: connect to target with instance 4 00000100:00080000:2.0:1767316261.107449:0:2489:0:(import.c:951:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff9d1f90a81800: Resetting ns_connect_flags to server flags: 0xa0425af2e3440038 00000080:00000004:2.0:1767316261.107460:0:2489:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0465af2e3640038 -> 0xa0425af2e3440038 00000080:00080000:2.0:1767316261.107466:0:2489:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1767316261.107474:0:2489:0:(import.c:1204:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:2.0:1767316261.107477:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f942fe800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:2.0:1767316261.107484:0:2489:0:(lov_obd.c:389:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:2.0:1767316261.107486:0:2489:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0425af2e3440038 -> 0xa0425af2e3440038 00000080:00080000:2.0:1767316261.107488:0:2489:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:2.0:1767316261.107499:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5b9cdf94-0180-42f8-9a71-beb807b227b6:2489:1853163904543616:192.168.202.140@tcp:8:mount.lustre.0 00000100:00100000:2.0:1767316261.109647:0:2489:0:(client.c:3081:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9d1f90a81800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1767316261.109665:0:2489:0:(import.c:1123:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff9d1f90a81800: connect to target with instance 2 00000100:00080000:2.0:1767316261.109673:0:2489:0:(import.c:951:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff9d1f90a81800: Resetting ns_connect_flags to server flags: 0xa0425af2e3440038 00000080:00000004:2.0:1767316261.109812:0:2489:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0425af2e3440038 -> 0xa0425af2e3440038 00000080:00080000:2.0:1767316261.109819:0:2489:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1767316261.109840:0:2489:0:(import.c:1204:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:2.0:1767316261.109844:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f942fd800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:2.0:1767316261.109851:0:2489:0:(lov_obd.c:389:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:2.0:1767316261.109855:0:2489:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0425af2e3440038 -> 0xa0425af2e3440038 00000080:00080000:2.0:1767316261.109858:0:2489:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:2.0:1767316261.109868:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5b9cdf94-0180-42f8-9a71-beb807b227b6:2489:1853163904543744:192.168.202.140@tcp:8:mount.lustre.0 00000100:00100000:3.0:1767316261.110483:0:46117:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:5b9cdf94-0180-42f8-9a71-beb807b227b6:46117:1853163904543872:192.168.202.140@tcp:33:mount.lustre.0 00000080:00200000:3.0:1767316261.110613:0:46117:0:(llite_lib.c:3217:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f885d3248) 00000080:00200000:3.0:1767316261.110621:0:46117:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:02000400:3.0:1767316261.110746:0:46117:0:(llite_lib.c:1533:ll_fill_super()) Mounted lustre-client 00000080:00000004:3.0:1767316261.112918:0:46117:0:(super25.c:202:lustre_fill_super()) 192.168.202.140@tcp:/lustre: Mount complete 00010000:00010000:3.0:1767316261.815953:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1767316262.288826:0:46154:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:3.0:1767316262.288852:0:46154:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316262.288871:0:46154:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316262.288887:0:46154:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316262.288895:0:46154:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1767316262.288903:0:46154:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:3.0:1767316262.288906:0:46154:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164910d1 00000080:00010000:3.0:1767316262.288913:0:46154:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164910d1 from it ffffb28141e4fb58 00010000:00010000:3.0:1767316262.288917:0:46154:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316262.288924:0:46154:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316262.288942:0:46154:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:3.0:1767316262.288956:0:46154:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=f203.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=8 00000080:00200000:3.0:1767316262.288960:0:46154:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=f203.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:3.0:1767316262.288973:0:46154:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f203.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316262.289021:0:46154:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46154 initiator: MDT0 00010000:00010000:3.0:1767316262.289035:0:46154:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46154 initiator: MDT0 00010000:00010000:3.0:1767316262.289046:0:46154:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46154 initiator: MDT0 00000100:00100000:3.0:1767316262.289076:0:46154:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job stat:45d767c3-7963-411a-be8f-bcbeb1c06075:46154:1853163904544000:192.168.202.140@tcp:101:stat.0 00000100:00100000:3.0:1767316262.289183:0:46154:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b000 going to sleep for 11 seconds 00000080:00200000:2.0:1767316262.787964:0:46157:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f885d3248),name=/ 00000002:00010000:2.0:1767316262.787999:0:46157:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316262.788073:0:46157:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.788085:0:46157:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.788096:0:46157:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46157 initiator: MDT0 00000100:00100000:2.0:1767316262.788124:0:46157:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904544128:192.168.202.140@tcp:101:lfs.0 00000100:00100000:2.0:1767316262.788286:0:46157:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f966f89c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316262.798787:0:46157:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904544128:192.168.202.140@tcp:101:lfs.0 00010000:00010000:2.0:1767316262.798825:0:46157:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.798840:0:46157:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.798852:0:46157:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.798867:0:46157:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.798903:0:46157:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.798912:0:46157:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.798919:0:46157:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316262.798931:0:46157:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e4380 x1853163904544128/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316278 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316262.798945:0:46157:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.798958:0:46157:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f88379c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1767316262.798965:0:46157:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316262.798996:0:46157:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1767316262.799001:0:46157:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f885d3248) for lock 0xa4df836b1649114f 00000080:00010000:2.0:1767316262.799007:0:46157:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f885d3248) 00000080:00010000:2.0:1767316262.799010:0:46157:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649114f from it ffffb28141e7fb58 00010000:00010000:2.0:1767316262.799024:0:46157:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.799033:0:46157:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316262.799058:0:46157:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f885d3248), inode mode 41ed mask 1 00000080:00200000:2.0:1767316262.799085:0:46157:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=f203.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f885d3248), flags=8 00000080:00200000:2.0:1767316262.799090:0:46157:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=f203.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f885d3248), intent=getattr 00000002:00010000:2.0:1767316262.799112:0:46157:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f203.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316262.799164:0:46157:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b16491156 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.799172:0:46157:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b16491156 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.799182:0:46157:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b16491156 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46157 initiator: MDT0 00000100:00100000:2.0:1767316262.799202:0:46157:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904544256:192.168.202.140@tcp:101:lfs.0 00000100:00100000:2.0:1767316262.799283:0:46157:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f966f89c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316262.811361:0:46157:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904544256:192.168.202.140@tcp:101:lfs.0 00010000:00010000:2.0:1767316262.811390:0:46157:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b16491156 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.811406:0:46157:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b16491156 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.811414:0:46157:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b16491156 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.811421:0:46157:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b16491156 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.811430:0:46157:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b16491156 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.811437:0:46157:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f909f3200 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:2.0:1767316262.811457:0:46157:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b16491156 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.811465:0:46157:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b16491156 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.811472:0:46157:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b16491156 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.811480:0:46157:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1767316262.811487:0:46157:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b16491156 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.811494:0:46157:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b16491156 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46157 initiator: MDT0 00000002:00100000:2.0:1767316262.811525:0:46157:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d1f890e4380 x1853163904544256/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 592/584 e 0 to 0 dl 1767316278 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1767316262.811542:0:46157:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1767316262.811570:0:46157:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.811583:0:46157:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00000080:00010000:2.0:1767316262.811592:0:46157:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649114f from it ffffb28141e7f940 00010000:00010000:2.0:1767316262.811596:0:46157:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.811603:0:46157:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316262.811787:0:46157:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f885d3248),name=/ 00000002:00010000:2.0:1767316262.811803:0:46157:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316262.811811:0:46157:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.811820:0:46157:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316262.811827:0:46157:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316262.811833:0:46157:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f885d3248) 00000080:00010000:2.0:1767316262.811837:0:46157:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f885d3248) for lock 0xa4df836b1649114f 00000080:00010000:2.0:1767316262.811842:0:46157:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649114f from it ffffb28141e7fbf8 00010000:00010000:2.0:1767316262.811846:0:46157:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.811854:0:46157:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316262.811864:0:46157:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f885d3248), inode mode 41ed mask 1 00000080:00200000:2.0:1767316262.811874:0:46157:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f203.replay-single, flags=777 00000080:00200000:2.0:1767316262.811891:0:46157:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f885d3248),name=/ 00000002:00010000:2.0:1767316262.811898:0:46157:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316262.811903:0:46157:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.811912:0:46157:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316262.811930:0:46157:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316262.811935:0:46157:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f885d3248) 00000080:00010000:2.0:1767316262.811938:0:46157:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f885d3248) for lock 0xa4df836b1649114f 00000080:00010000:2.0:1767316262.811942:0:46157:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649114f from it ffffb28141e7fb38 00010000:00010000:2.0:1767316262.811946:0:46157:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.811954:0:46157:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316262.811963:0:46157:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f885d3248), inode mode 41ed mask 3 00000080:00200000:2.0:1767316262.811969:0:46157:0:(namei.c:1354:ll_atomic_open()) VFS Op:name=f203.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f885d3248), file ffff9d1f88109e00, open_flags a141, mode 81b6 opened 0 00000080:00200000:2.0:1767316262.811978:0:46157:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=f203.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f885d3248), intent=open|creat 00000100:00100000:2.0:1767316262.812048:0:46157:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904544384:192.168.202.140@tcp:700:lfs.0 00000100:00100000:2.0:1767316262.812150:0:46157:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f966f89c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316262.825170:0:46157:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904544384:192.168.202.140@tcp:700:lfs.0 00000002:00010000:2.0:1767316262.825453:0:46157:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f203.replay-single,[0x2000059f1:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0120502 00010000:00010000:2.0:1767316262.825476:0:46157:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.825551:0:46157:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.825559:0:46157:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316262.825562:0:46157:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3c00/0xa4df836b1649114f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9781 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.825629:0:46157:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 46157 initiator: MDT0 00010000:00010000:2.0:1767316262.825636:0:46157:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46157 initiator: MDT0 00000020:00100000:2.0:1767316262.825648:0:46157:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f90a81800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1767316262.825657:0:46157:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4380 x1853163904544640/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 536/66432 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1767316262.825668:0:46157:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46157 initiator: MDT0 00000100:00100000:2.0:1767316262.825749:0:46157:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904544640:192.168.202.140@tcp:101:lfs.0 00000100:00100000:2.0:1767316262.825893:0:46157:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f966f89c0 going to sleep for 11 seconds 00000100:00100000:0.0:1767316262.839852:0:2482:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.140@tcp (source: 12345-192.168.202.140@tcp) 00000100:00100000:1.0:1767316262.840318:0:13861:0:(service.c:2279:ptlrpc_server_handle_req_in()) unwrap req x1853163946290688 00000100:00100000:1.0:1767316262.840329:0:13861:0:(service.c:2344:ptlrpc_server_handle_req_in()) got req x1853163946290688 00000100:00100000:1.0:1767316262.840380:0:13861:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.140@tcp, seq: 1431 00000100:00100000:1.0:1767316262.840390:0:13861:0:(service.c:2530:ptlrpc_server_handle_request()) Handling RPC req@ffff9d1f91033480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:45d767c3-7963-411a-be8f-bcbeb1c06075+5:51369:x1853163946290688:12345-192.168.202.140@tcp:104: 00010000:00010000:1.0:1767316262.840422:0:13861:0:(ldlm_lockd.c:2505:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316262.840506:0:13861:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1767316262.840523:0:13861:0:(service.c:2588:ptlrpc_server_handle_request()) Handled RPC req@ffff9d1f91033480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:45d767c3-7963-411a-be8f-bcbeb1c06075+5:51369:x1853163946290688:12345-192.168.202.140@tcp:104: Request processed in 135us (685us total) trans 0 rc 0/0 00000100:00100000:1.0:1767316262.840530:0:13861:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.140@tcp, seq: 1431 00010000:00010000:2.0:1767316262.840559:0:15046:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1767316262.840568:0:15046:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.840578:0:15046:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f93ccc000 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:2.0:1767316262.840584:0:15046:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.840594:0:15046:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.840603:0:15046:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.840661:0:15046:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.840675:0:15046:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316262.840737:0:15046:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.840745:0:15046:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccc000/0xa4df836b164910d1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xcb3a17585d8c96ee expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.840764:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316262.840766:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316262.840768:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767316262.841226:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904544768:192.168.202.140@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1767316262.842159:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767316262.873098:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904544768:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1767316262.915776:0:46157:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904544640:192.168.202.140@tcp:101:lfs.0 00010000:00010000:2.0:1767316262.915807:0:46157:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316262.918081:0:46157:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 4/0,1 mode: --/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316262.918100:0:46157:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 4/0,1 mode: --/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316262.918113:0:46157:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 4/0,1 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316262.918143:0:46157:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 4/0,1 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316262.918164:0:46157:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 5/1,1 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316262.918844:0:46157:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 5/1,1 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316262.918878:0:46157:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 4/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1767316262.918908:0:46157:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=8000037, status=0 req@ffff9d1f890e4380 x1853163904544640/t51539607555(51539607555) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 536/608 e 0 to 0 dl 1767316278 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1767316262.918922:0:46157:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4380 x1853163904544640/t51539607555(51539607555) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 536/608 e 0 to 0 dl 1767316278 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1767316262.918936:0:46157:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 3/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316262.918947:0:46157:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f88379c00 type 13 mode 4 res 8589957617/1 (0 0) 00000080:00200000:3.0:1767316262.918953:0:46157:0:(namei.c:955:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1767316262.918968:0:46157:0:(namei.c:1569:ll_create_it()) VFS Op:name=f203.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f885d3248), intent=open|creat 00000080:00200000:3.0:1767316262.919002:0:46157:0:(llite_lib.c:3217:ll_read_inode2()) VFS Op:inode=[0x2000059f1:0x1:0x0](ffff9d1f885d6c08) 00000080:00200000:3.0:1767316262.919009:0:46157:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x2000059f1:0x1:0x0], updating i_size 0 00000080:00200000:3.0:1767316262.919044:0:46157:0:(file.c:6809:ll_layout_conf()) [0x2000059f1:0x1:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1767316262.919049:0:46157:0:(namei.c:1535:ll_create_node()) setting l_ast_data to inode [0x2000059f1:0x1:0x0](ffff9d1f885d6c08) 00000080:00010000:3.0:1767316262.919053:0:46157:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x2000059f1:0x1:0x0](ffff9d1f885d6c08) for lock 0xa4df836b1649115d 00000080:00010000:3.0:1767316262.919067:0:46157:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000059f1:0x1:0x0](ffff9d1f885d6c08) 00000080:00200008:3.0:1767316262.919089:0:46157:0:(file.c:1001:ll_file_open()) START file f203.replay-single:[0x2000059f1:0x1:0x0](ffff9d1f885d6c08), flags 120501 00000002:00100000:3.0:1767316262.919110:0:46157:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4380 x1853163904544640/t51539607555(51539607555) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 536/608 e 0 to 0 dl 1767316278 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1767316262.919148:0:46157:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649115d from it ffff9d1f918ce2a0 00010000:00010000:3.0:1767316262.919153:0:46157:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 3/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316262.919171:0:46157:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 2/0,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316262.919214:0:46157:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x2000059f1:0x1:0x0](ffff9d1f885d6c08), inode mode 81a4 mask 40 00000080:00200000:3.0:1767316262.920652:0:46157:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x2000059f1:0x1:0x0](ffff9d1f885d6c08), inode mode 81a4 mask 2 00000080:00200000:3.0:1767316262.920671:0:46157:0:(xattr.c:406:ll_xattr_set()) VFS Op:inode=[0x2000059f1:0x1:0x0](ffff9d1f885d6c08), xattr lov 00000002:00010000:3.0:1767316262.920713:0:46157:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000059f1:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 040000002 00010000:00010000:3.0:1767316262.920790:0:46157:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f8b384e00/0xa4df836b16491164 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 46157 initiator: MDT0 00010000:00010000:3.0:1767316262.920800:0:46157:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f8b384e00/0xa4df836b16491164 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46157 initiator: MDT0 00000020:00100000:3.0:1767316262.920813:0:46157:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f90a81800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1767316262.920824:0:46157:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f92f73b80 x1853163904545024/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 1512/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1767316262.920834:0:46157:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f8b384e00/0xa4df836b16491164 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46157 initiator: MDT0 00000100:00100000:3.0:1767316262.920859:0:46157:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73b80 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904545024:192.168.202.140@tcp:101:lfs.0 00000100:00100000:3.0:1767316262.922627:0:46157:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1767316262.925549:0:2483:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.140@tcp (source: 12345-192.168.202.140@tcp) 00000100:00100000:0.0:1767316262.925613:0:7197:0:(service.c:2279:ptlrpc_server_handle_req_in()) unwrap req x1853163946290816 00000100:00100000:0.0:1767316262.925624:0:7197:0:(service.c:2344:ptlrpc_server_handle_req_in()) got req x1853163946290816 00000100:00100000:0.0:1767316262.925673:0:7197:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.140@tcp, seq: 1432 00000100:00100000:0.0:1767316262.925724:0:7197:0:(service.c:2530:ptlrpc_server_handle_request()) Handling RPC req@ffff9d1f91030700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:5b9cdf94-0180-42f8-9a71-beb807b227b6+5:51369:x1853163946290816:12345-192.168.202.140@tcp:104: 00010000:00010000:0.0:1767316262.925741:0:7197:0:(ldlm_lockd.c:2505:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 2/0,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316262.925809:0:7197:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1767316262.925815:0:7197:0:(service.c:2588:ptlrpc_server_handle_request()) Handled RPC req@ffff9d1f91030700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:5b9cdf94-0180-42f8-9a71-beb807b227b6+5:51369:x1853163946290816:12345-192.168.202.140@tcp:104: Request processed in 95us (279us total) trans 0 rc 0/0 00000100:00100000:0.0:1767316262.925823:0:7197:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.140@tcp, seq: 1432 00010000:00010000:2.0:1767316262.925998:0:7207:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1767316262.926007:0:7207:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 2/0,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.926017:0:7207:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f909f3200 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:2.0:1767316262.926022:0:7207:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 2/0,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.926030:0:7207:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 2/0,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.926038:0:7207:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 3/0,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316262.926081:0:7207:0:(vvp_object.c:114:vvp_conf_set()) [0x2000059f1:0x1:0x0]: losing layout lock 00010000:00010000:2.0:1767316262.926127:0:7207:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 2/0,0 mode: --/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.926135:0:7207:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316262.926163:0:7207:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 1/0,0 mode: --/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.926172:0:7207:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f909f3200/0xa4df836b1649115d lrc: 0/0,0 mode: --/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xcb3a17585d8c97b2 expref: -99 pid: 46157 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316262.926198:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316262.926200:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316262.926202:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316262.926210:0:2493:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1767316262.926384:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904545152:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1767316262.943774:0:46157:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73b80 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904545024:192.168.202.140@tcp:101:lfs.0 00010000:00010000:3.0:1767316262.943797:0:46157:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f8b384e00/0xa4df836b16491164 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46157 initiator: MDT0 00010000:00010000:3.0:1767316262.943810:0:46157:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f8b384e00/0xa4df836b16491164 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 46157 initiator: MDT0 00010000:00010000:3.0:1767316262.943838:0:46157:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f8b384e00/0xa4df836b16491164 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 46157 initiator: MDT0 00010000:00010000:3.0:1767316262.943846:0:46157:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f8b384e00/0xa4df836b16491164 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 46157 initiator: MDT0 00010000:00010000:3.0:1767316262.943854:0:46157:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f8b384e00/0xa4df836b16491164 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 46157 initiator: MDT0 00010000:00010000:3.0:1767316262.943861:0:46157:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f8b384e00 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:3.0:1767316262.943867:0:46157:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f8b384e00/0xa4df836b16491164 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 46157 initiator: MDT0 00010000:00010000:3.0:1767316262.943875:0:46157:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f8b384e00/0xa4df836b16491164 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 46157 initiator: MDT0 00010000:00010000:3.0:1767316262.943882:0:46157:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f8b384e00/0xa4df836b16491164 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 46157 initiator: MDT0 00010000:00010000:3.0:1767316262.943889:0:46157:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1767316262.943896:0:46157:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f8b384e00/0xa4df836b16491164 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 46157 initiator: MDT0 00010000:00010000:3.0:1767316262.943902:0:46157:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f8b384e00/0xa4df836b16491164 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 46157 initiator: MDT0 00000002:00100000:3.0:1767316262.943931:0:46157:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=800002b, status=0 req@ffff9d1f92f73b80 x1853163904545024/t51539607556(51539607556) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 1512/1912 e 0 to 0 dl 1767316278 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1767316262.943946:0:46157:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f92f73b80 x1853163904545024/t51539607556(51539607556) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 1512/1912 e 0 to 0 dl 1767316278 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1767316262.943965:0:46157:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1767316262.944052:0:46157:0:(vvp_io.c:1854:vvp_io_init()) [0x2000059f1:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1767316262.944061:0:46157:0:(vvp_io.c:289:vvp_io_fini()) [0x2000059f1:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1767316262.944128:0:46157:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x2000059f1:0x1:0x0], updating i_size 0 00000002:00100000:3.0:1767316262.944146:0:46157:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f92f73b80 x1853163904545024/t51539607556(51539607556) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 1824/1912 e 0 to 0 dl 1767316278 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1767316262.944190:0:46157:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f92f73b80 x1853163904545024/t51539607556(51539607556) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 1824/1912 e 0 to 0 dl 1767316278 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1767316262.944201:0:46157:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f90a81800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1767316262.944206:0:46157:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f92f70000 x1853163904545408/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1767316262.944239:0:46157:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904545408:192.168.202.140@tcp:35:lfs.0 00000100:00100000:3.0:1767316262.944325:0:46157:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316262.944561:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904545152:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1767316262.946908:0:46157:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904545408:192.168.202.140@tcp:35:lfs.0 00000002:00100000:3.0:1767316262.946948:0:46157:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f92f73b80 x1853163904545024/t51539607556(51539607556) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 1824/1912 e 0 to 0 dl 1767316278 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1767316262.947012:0:46157:0:(file.c:424:ll_file_release()) START file f203.replay-single:[0x2000059f1:0x1:0x0](ffff9d1f885d6c08), flags 100001 00000002:00100000:3.0:1767316262.947066:0:46157:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f890e4380 x1853163904544640/t51539607555(51539607555) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 536/608 e 0 to 0 dl 1767316278 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1767316262.947077:0:46157:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f90a81800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1767316262.947083:0:46157:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f92f71500 x1853163904545664/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1767316262.947106:0:46157:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904545664:192.168.202.140@tcp:35:lfs.0 00000100:00100000:3.0:1767316262.947194:0:46157:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1767316262.948963:0:46157:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job lfs:5b9cdf94-0180-42f8-9a71-beb807b227b6:46157:1853163904545664:192.168.202.140@tcp:35:lfs.0 00000002:00100000:3.0:1767316262.949009:0:46157:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f890e4380 x1853163904544640/t51539607555(51539607555) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 536/608 e 0 to 0 dl 1767316278 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1767316263.865894:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316264.328851:0:2482:0:(events.c:108:reply_in_callback()) truncate req ffff9d1f92f71880 rpc 1376 - 1888+224 00000100:00100000:3.0:1767316264.329035:0:46154:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b000 going to sleep for 1 seconds 00000100:00080000:3.0:1767316264.329053:0:46154:0:(client.c:3710:ptlrpc_set_mbits()) resend with new mbits old x1853163904544000 new x1853163904545792 00000100:00100000:3.0:1767316264.329170:0:46154:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b000 going to sleep for 11 seconds 00000100:00100000:3.0:1767316264.335640:0:46154:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job stat:45d767c3-7963-411a-be8f-bcbeb1c06075:46154:1853163904544000:192.168.202.140@tcp:101:stat.0 00010000:00010000:3.0:1767316264.335669:0:46154:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316264.335772:0:46154:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 4/1,0 mode: --/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316264.335781:0:46154:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 4/1,0 mode: --/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316264.335792:0:46154:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 4/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316264.335803:0:46154:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 4/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316264.335814:0:46154:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 5/2,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316264.335822:0:46154:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 5/2,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316264.335828:0:46154:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 4/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1767316264.335856:0:46154:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f92f71880 x1853163904544000/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/1888 e 0 to 0 dl 1767316280 ref 1 fl Complete:RQU/602/0 rc 0/0 job:'stat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1767316264.335868:0:46154:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 1304 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 3/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 0 00000002:00010000:3.0:1767316264.335881:0:46154:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 3/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316264.335890:0:46154:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589957617/1 (0 0) 00000080:00200000:3.0:1767316264.335898:0:46154:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1767316264.335928:0:46154:0:(llite_lib.c:3217:ll_read_inode2()) VFS Op:inode=[0x2000059f1:0x1:0x0](ffff9d1f885d7448) 00000080:00200000:3.0:1767316264.336020:0:46154:0:(file.c:6809:ll_layout_conf()) [0x2000059f1:0x1:0x0]: layout version change: 4294967294 -> 12 00000080:00010000:3.0:1767316264.336026:0:46154:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x2000059f1:0x1:0x0](ffff9d1f885d7448) for lock 0xa4df836b16491148 00000080:00010000:3.0:1767316264.336038:0:46154:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000059f1:0x1:0x0](ffff9d1f885d7448) 00000080:00010000:3.0:1767316264.336042:0:46154:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491148 from it ffffb28141e4fb20 00010000:00010000:3.0:1767316264.336045:0:46154:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 3/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316264.336053:0:46154:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 2/0,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00000080:00200008:3.0:1767316264.336093:0:46154:0:(file.c:6252:ll_getattr_dentry()) START file f203.replay-single:[0x2000059f1:0x1:0x0](ffff9d1f885d7448), request_mask 4095, flags 0, foreign 0 00000080:00200000:3.0:1767316264.336100:0:46154:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x2000059f1:0x1:0x0](ffff9d1f885d7448),name=f203.replay-single 00000002:00010000:3.0:1767316264.336199:0:46154:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000059f1:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316264.336218:0:46154:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 2/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316264.336241:0:46154:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 2/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1767316264.336247:0:46154:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1767316264.336252:0:46154:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000059f1:0x1:0x0](ffff9d1f885d7448) 00000080:00010000:3.0:1767316264.336256:0:46154:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x2000059f1:0x1:0x0](ffff9d1f885d7448) for lock 0xa4df836b16491148 00000080:00010000:3.0:1767316264.336260:0:46154:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491148 from it ffffb28141e4fcb8 00010000:00010000:3.0:1767316264.336263:0:46154:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 3/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316264.336270:0:46154:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 2/0,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1767316264.336289:0:46154:0:(vvp_io.c:1854:vvp_io_init()) [0x2000059f1:0x1:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1767316264.336299:0:46154:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x2000059f1:0x1:0x0] 00000008:00010000:3.0:1767316264.336315:0:46154:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1767316264.336318:0:46154:0:(osc_lock.c:1248:osc_lock_init()) ### lock ffff9d1f93c82f30, osc lock ffff9d1f8af13430, flags 0x400001002 00010000:00010000:3.0:1767316264.336365:0:46154:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 3/1,0 mode: --/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1767316264.336373:0:46154:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 3/1,0 mode: --/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->1048575] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1767316264.336578:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904545920:192.168.202.140@tcp:101:stat.0 00000020:00200000:3.0:1767316264.336585:0:46154:0:(cl_io.c:1802:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1767316264.341381:0:2493:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->1048575] (req 0->1048575) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316264.341422:0:2493:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316264.341438:0:2493:0:(ldlm_resource.c:1525:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316264.341473:0:2493:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00000008:00010000:2.0:1767316264.341490:0:2493:0:(osc_lock.c:179:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00000020:00200000:2.0:1767316264.341512:0:2493:0:(cl_io.c:1802:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1767316264.341532:0:2493:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316264.341540:0:2493:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316264.341547:0:2493:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316264.341555:0:2493:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1767316264.341577:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904545920:192.168.202.140@tcp:101:stat.0 00000080:00200000:3.0:1767316264.341765:0:46154:0:(file.c:1686:ll_merge_attr_nolock()) [0x2000059f1:0x1:0x0] updating i_size 0 i_blocks 1 00000020:00010000:3.0:1767316264.341773:0:46154:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d1f88d26480 (ffff9d1f8922aab8/0) at cl_lock_release():245 00000020:00010000:3.0:1767316264.341777:0:46154:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d1f88d26480 (ffff9d1f8922aab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1767316264.341783:0:46154:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d1f93c82f30 (ffff9d1f8922a878/1) at cl_lock_cancel():142 00000020:00010000:3.0:1767316264.341796:0:46154:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d1f88d26480 (ffff9d1f8922aab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1767316264.341799:0:46154:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d1f93c82f30 (ffff9d1f8922aab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1767316264.341815:0:46154:0:(vvp_io.c:289:vvp_io_fini()) [0x2000059f1:0x1:0x0] ignore/verify layout 0/1, layout version 12 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1767316264.886851:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316265.914818:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316266.296123:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904546176:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316266.296221:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904546432:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316266.296246:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904546688:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316266.296285:0:2493:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [0->1] 00000100:00100000:2.0:1767316266.296292:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904546048:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316266.296331:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904546304:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316266.296355:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904546560:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316266.296376:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904546816:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316266.310342:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904546304:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316266.310364:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904546432:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316266.310384:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904546688:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316266.311546:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904546048:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316266.324213:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904546560:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316266.324238:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904546176:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316266.338665:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904546816:192.168.202.140@tcp:400:kworker.0 00010000:00010000:2.0:1767316266.934823:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316267.962760:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316268.984395:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316270.013723:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1767316270.097758:0:46593:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1767316270.097774:0:46593:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316270.097799:0:46593:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316270.097862:0:46593:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 46593 initiator: MDT0 00010000:00010000:1.0:1767316270.097872:0:46593:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46593 initiator: MDT0 00010000:00010000:1.0:1767316270.097881:0:46593:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46593 initiator: MDT0 00000100:00100000:1.0:1767316270.097907:0:46593:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job lsof:45d767c3-7963-411a-be8f-bcbeb1c06075:46593:1853163904546944:192.168.202.140@tcp:101:lsof.0 00000100:00100000:1.0:1767316270.098009:0:46593:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a4d840 going to sleep for 11 seconds 00000100:00100000:0.0:1767316270.101003:0:46593:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job lsof:45d767c3-7963-411a-be8f-bcbeb1c06075:46593:1853163904546944:192.168.202.140@tcp:101:lsof.0 00010000:00010000:0.0:1767316270.101028:0:46593:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.101040:0:46593:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.101051:0:46593:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.101062:0:46593:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.101073:0:46593:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.101080:0:46593:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.101101:0:46593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316270.101112:0:46593:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f918d3b80 x1853163904546944/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316286 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lsof.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316270.101125:0:46593:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.101138:0:46593:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1767316270.101144:0:46593:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1767316270.101170:0:46593:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1767316270.101177:0:46593:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491172 00000080:00010000:0.0:1767316270.101183:0:46593:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316270.101186:0:46593:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491172 from it ffffb28141c47cb8 00010000:00010000:0.0:1767316270.101189:0:46593:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.101197:0:46593:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200008:0.0:1767316270.103861:0:46593:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f885d3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1767316270.103875:0:46593:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f885d3248),name=/ 00000002:00010000:0.0:1767316270.103899:0:46593:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316270.103959:0:46593:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 46593 initiator: MDT0 00010000:00010000:0.0:1767316270.103969:0:46593:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46593 initiator: MDT0 00010000:00010000:0.0:1767316270.103986:0:46593:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46593 initiator: MDT0 00000100:00100000:0.0:1767316270.104010:0:46593:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job lsof:5b9cdf94-0180-42f8-9a71-beb807b227b6:46593:1853163904547072:192.168.202.140@tcp:101:lsof.0 00000100:00100000:0.0:1767316270.104139:0:46593:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f88daa840 going to sleep for 11 seconds 00000100:00100000:0.0:1767316270.107888:0:46593:0:(client.c:3081:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d1f90a81800: committing for last_committed 51539607558 gen 1 00000100:00100000:0.0:1767316270.107983:0:46593:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job lsof:5b9cdf94-0180-42f8-9a71-beb807b227b6:46593:1853163904547072:192.168.202.140@tcp:101:lsof.0 00010000:00010000:0.0:1767316270.108009:0:46593:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.108022:0:46593:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.108035:0:46593:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.108053:0:46593:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.108064:0:46593:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.108072:0:46593:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.108079:0:46593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316270.108106:0:46593:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f91030380 x1853163904547072/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316286 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lsof.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316270.108120:0:46593:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.108134:0:46593:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f88379c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1767316270.108141:0:46593:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1767316270.108167:0:46593:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1767316270.108173:0:46593:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f885d3248) for lock 0xa4df836b16491179 00000080:00010000:0.0:1767316270.108179:0:46593:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f885d3248) 00000080:00010000:0.0:1767316270.108182:0:46593:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491179 from it ffffb28141c47cb8 00010000:00010000:0.0:1767316270.108185:0:46593:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316270.108193:0:46593:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316271.032964:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767316271.424737:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904547200:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316271.424843:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904547456:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316271.424872:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904547712:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316271.424893:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904547968:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316271.424920:0:2493:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1767316271.424926:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904547328:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316271.424959:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904547584:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316271.424979:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904547840:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316271.428876:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904547328:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316271.428893:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904547584:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316271.428902:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904547840:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316271.428911:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904547200:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316271.428920:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904547456:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316271.428929:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904547712:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316271.428938:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:5b9cdf94-0180-42f8-9a71-beb807b227b6:2493:1853163904547968:192.168.202.140@tcp:400:kworker.0 00000080:00200008:2.0:1767316271.989896:0:46601:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f885d3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316271.989915:0:46601:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f885d3248),name=/ 00000002:00010000:2.0:1767316271.989940:0:46601:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316271.989958:0:46601:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316271.989975:0:46601:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316271.989983:0:46601:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316271.989991:0:46601:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f885d3248) 00000080:00010000:2.0:1767316271.989995:0:46601:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f885d3248) for lock 0xa4df836b16491179 00000080:00010000:2.0:1767316271.990002:0:46601:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491179 from it ffffb28141cdfcb8 00010000:00010000:2.0:1767316271.990007:0:46601:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316271.990014:0:46601:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200008:2.0:1767316271.990225:0:46601:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f885d3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316271.990231:0:46601:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f885d3248),name=/ 00000002:00010000:2.0:1767316271.990239:0:46601:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316271.990246:0:46601:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316271.990254:0:46601:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316271.990261:0:46601:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316271.990265:0:46601:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f885d3248) 00000080:00010000:2.0:1767316271.990268:0:46601:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f885d3248) for lock 0xa4df836b16491179 00000080:00010000:2.0:1767316271.990272:0:46601:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491179 from it ffffb28141cdfcb8 00010000:00010000:2.0:1767316271.990275:0:46601:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316271.990282:0:46601:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316272.009865:0:46601:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f885d3248) 00000080:00200000:3.0:1767316272.009922:0:46601:0:(vvp_io.c:1854:vvp_io_init()) [0x2000059f1:0x1:0x0] ignore/verify layout 1/0, layout version 12 restore needed 0 00020000:00200000:3.0:1767316272.009936:0:46601:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:3.0:1767316272.009946:0:46601:0:(lov_io.c:860:lov_io_iter_init()) component[1] flags 0x0 00020000:00200000:3.0:1767316272.009947:0:46601:0:(lov_io.c:860:lov_io_iter_init()) component[2] flags 0x0 00020000:00200000:3.0:1767316272.009949:0:46601:0:(lov_io.c:860:lov_io_iter_init()) component[3] flags 0x0 00020000:00200000:3.0:1767316272.009950:0:46601:0:(lov_io.c:860:lov_io_iter_init()) component[4] flags 0x0 00020000:00200000:3.0:1767316272.009951:0:46601:0:(lov_io.c:860:lov_io_iter_init()) component[5] flags 0x0 00020000:00200000:3.0:1767316272.009952:0:46601:0:(lov_io.c:860:lov_io_iter_init()) component[6] flags 0x0 00020000:00200000:3.0:1767316272.009954:0:46601:0:(lov_io.c:860:lov_io_iter_init()) component[7] flags 0x0 00020000:00200000:3.0:1767316272.009955:0:46601:0:(lov_io.c:860:lov_io_iter_init()) component[8] flags 0x0 00020000:00200000:3.0:1767316272.009956:0:46601:0:(lov_io.c:860:lov_io_iter_init()) component[9] flags 0x0 00020000:00200000:3.0:1767316272.009958:0:46601:0:(lov_io.c:860:lov_io_iter_init()) component[10] flags 0x0 00020000:00200000:3.0:1767316272.009959:0:46601:0:(lov_io.c:860:lov_io_iter_init()) component[11] flags 0x0 00000080:00200000:3.0:1767316272.009981:0:46601:0:(vvp_io.c:289:vvp_io_fini()) [0x2000059f1:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1767316272.010008:0:46601:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x2000059f1:0x1:0x0](ffff9d1f885d6c08) 00000080:00200000:3.0:1767316272.010138:0:46601:0:(llite_lib.c:1568:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff9d1f90a81800 (sb ffff9d1f90a81800) 00010000:00010000:3.0:1767316272.010464:0:46601:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316272.010476:0:46601:0:(ldlm_lock.c:947:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000000000000 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316272.010484:0:46601:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000400000000 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316272.010490:0:46601:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000400000000 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316272.010519:0:46601:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1767316272.010526:0:46601:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316272.010534:0:46601:0:(ldlm_lock.c:947:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316272.010539:0:46601:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316272.010545:0:46601:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316272.010565:0:46601:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:3.0:1767316272.010572:0:46601:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316272.010579:0:46601:0:(ldlm_lock.c:947:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316272.010591:0:46601:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316272.010597:0:46601:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316272.010617:0:46601:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/3 locks to regular blp list, 3 blwis in pool 10000000:01000000:3.0:1767316272.010621:0:46601:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00020000:00020000:3.0:1767316272.010984:0:46601:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9d1f90a81800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 00010000:00010000:0.0:1767316272.011155:0:7208:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:0.0:1767316272.011163:0:7208:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000400000000 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011171:0:7208:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f93c84600 already unused, calling callback (ffffffffc1141c20) 10000000:00010000:0.0:1767316272.011174:0:7208:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000400000000 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011182:0:7208:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 10000000:00010000:0.0:1767316272.011201:0:7208:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 10000000:01000000:0.0:1767316272.011206:0:7208:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:0.0:1767316272.011209:0:7208:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 10000000:01000000:0.0:1767316272.011211:0:7208:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 00010000:00010000:0.0:1767316272.011250:0:7208:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011256:0:7208:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1767316272.011279:0:7208:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011284:0:7208:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84600/0xa4df836b164910ed lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xcb3a17585d8c9749 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011295:0:7208:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1767316272.011299:0:7208:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011304:0:7208:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f93c84000 already unused, calling callback (ffffffffc1141c20) 10000000:00010000:0.0:1767316272.011306:0:7208:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011310:0:7208:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 10000000:00010000:0.0:1767316272.011316:0:7208:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 10000000:01000000:0.0:1767316272.011321:0:7208:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:0.0:1767316272.011340:0:7208:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:0.0:1767316272.011342:0:7208:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:0.0:1767316272.011355:0:7208:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011360:0:7208:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1767316272.011365:0:7208:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011370:0:7208:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84000/0xa4df836b16491117 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xcb3a17585d8c9750 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011379:0:7208:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1767316272.011382:0:7208:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011387:0:7208:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f93c84a00 already unused, calling callback (ffffffffc1141c20) 10000000:00010000:0.0:1767316272.011389:0:7208:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011394:0:7208:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 10000000:00010000:0.0:1767316272.011400:0:7208:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 10000000:01000000:0.0:1767316272.011404:0:7208:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:0.0:1767316272.011405:0:7208:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:0.0:1767316272.011408:0:7208:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:0.0:1767316272.011417:0:7208:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011422:0:7208:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1767316272.011444:0:7208:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011449:0:7208:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.140@tcp lock: ffff9d1f93c84a00/0xa4df836b1649111e lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xcb3a17585d8c9757 expref: -99 pid: 46117 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.011459:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767316272.011460:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767316272.011462:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1767316272.011483:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904548096:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1767316272.011575:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904548352:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1767316272.011600:0:2490:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1767316272.011605:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904548224:192.168.202.140@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1767316272.011918:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316272.011973:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00020000:01000000:3.0:1767316272.024773:0:46601:0:(lov_obd.c:589:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff9d1f8af58000 active: 1 00020000:00020000:3.0:1767316272.024785:0:46601:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9d1f90a81800: lov tgt 1 not cleaned! deathrow=1, lovrc=1 00020000:01000000:3.0:1767316272.024788:0:46601:0:(lov_obd.c:589:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff9d1f8af5e000 active: 1 00020000:01000000:3.0:1767316272.024790:0:46601:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:3.0:1767316272.024796:0:46601:0:(lov_obd.c:611:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff9d1f90a81800 00020000:01000000:3.0:1767316272.024797:0:46601:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff9d1f90a81800: disconnecting target lustre-OST0001-osc-ffff9d1f90a81800 00000100:00080000:3.0:1767316272.024804:0:46601:0:(pinger.c:396:ptlrpc_pinger_del_import()) removing pingable import 5b9cdf94-0180-42f8-9a71-beb807b227b6->lustre-OST0001_UUID 00000100:00080000:3.0:1767316272.024931:0:46601:0:(import.c:65:import_set_state_nolock()) ffff9d1f942fd800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1767316272.025321:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5b9cdf94-0180-42f8-9a71-beb807b227b6:2489:1853163904548480:192.168.202.140@tcp:9:umount.0 00000100:00100000:0.0:1767316272.036977:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904548224:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1767316272.036996:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904548096:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1767316272.037004:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2490:1853163904548352:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00080000:2.0:1767316272.048663:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f942fd800 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1767316272.048732:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5b9cdf94-0180-42f8-9a71-beb807b227b6:2489:1853163904548480:192.168.202.140@tcp:9:umount.0 00000100:00080000:3.0:1767316272.048818:0:46601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:3.0:1767316272.048822:0:46601:0:(client.c:3081:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9d1f90a81800: committing for last_committed 0 gen 2 00000100:00100000:3.0:1767316272.048827:0:46601:0:(import.c:321:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1767316272.048983:0:46601:0:(genops.c:1485:class_disconnect()) disconnect: cookie 0xa4df836b16491141 00000020:00000080:3.0:1767316272.049002:0:46601:0:(genops.c:904:class_export_put()) final put ffff9d1f8af5e000/LOV_OSC_UUID 00000020:00000080:1.0:1767316272.049064:0:42605:0:(genops.c:863:class_export_destroy()) destroying export ffff9d1f8af5e000/LOV_OSC_UUID for lustre-OST0001-osc-ffff9d1f90a81800 00000020:01000000:3.0:1767316272.049069:0:46601:0:(obd_config.c:2361:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff9d1f90a81800 (flags='') 00000020:00000080:3.0:1767316272.049075:0:46601:0:(obd_config.c:1389:class_process_config()) processing cmd: cf004 00000020:00000080:3.0:1767316272.049085:0:46601:0:(obd_config.c:886:class_cleanup()) lustre-OST0001-osc-ffff9d1f90a81800: forcing exports to disconnect: 0/2 00000020:00080000:3.0:1767316272.049089:0:46601:0:(genops.c:1575:class_disconnect_exports()) OBD device 9 (ffff9d1f8b39b1b0) has no exports 00000008:01000000:3.0:1767316272.049124:0:46601:0:(obd_class.h:663:obd_cleanup_client_import()) lustre-OST0001-osc-ffff9d1f90a81800: client import never connected 00000100:00080000:3.0:1767316272.049127:0:46601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:3.0:1767316272.049145:0:46601:0:(client.c:3081:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9d1f90a81800: committing for last_committed 0 gen 3 00000100:00020000:3.0:1767316272.049148:0:46601:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup 00000100:00100000:3.0:1767316272.053564:0:46601:0:(import.c:321:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000100:00020000:3.0:1767316272.057738:0:46601:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup 00000020:00000080:1.0:1767316272.058015:0:42605:0:(genops.c:1202:obd_zombie_import_free()) destroying import ffff9d1f942fd800 for lustre-OST0001-osc-ffff9d1f90a81800 00000020:00000080:3.0:1767316272.058052:0:46601:0:(obd_config.c:1389:class_process_config()) processing cmd: cf002 00000020:00000080:3.0:1767316272.058065:0:46601:0:(obd_config.c:814:class_detach()) detach on obd lustre-OST0001-osc-ffff9d1f90a81800 (uuid 5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000020:00000080:3.0:1767316272.058071:0:46601:0:(genops.c:904:class_export_put()) final put ffff9d1f942fe000/5b9cdf94-0180-42f8-9a71-beb807b227b6 00000020:00000080:3.0:1767316272.058074:0:46601:0:(genops.c:863:class_export_destroy()) destroying export ffff9d1f942fe000/5b9cdf94-0180-42f8-9a71-beb807b227b6 for lustre-OST0001-osc-ffff9d1f90a81800 00000020:01000000:3.0:1767316272.058079:0:46601:0:(genops.c:443:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff9d1f90a81800 (5b9cdf94-0180-42f8-9a71-beb807b227b6) 00020000:01000000:3.0:1767316272.058269:0:46601:0:(lov_obd.c:611:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff9d1f90a81800 00020000:01000000:3.0:1767316272.058272:0:46601:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff9d1f90a81800: disconnecting target lustre-OST0000-osc-ffff9d1f90a81800 00000100:00080000:3.0:1767316272.058277:0:46601:0:(pinger.c:396:ptlrpc_pinger_del_import()) removing pingable import 5b9cdf94-0180-42f8-9a71-beb807b227b6->lustre-OST0000_UUID 00000100:00080000:3.0:1767316272.058401:0:46601:0:(import.c:65:import_set_state_nolock()) ffff9d1f942fe800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:0.0:1767316272.058860:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5b9cdf94-0180-42f8-9a71-beb807b227b6:2489:1853163904548608:192.168.202.140@tcp:9:umount.0 00010000:00010000:0.0:1767316272.059434:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1767316272.064353:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f942fe800 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1767316272.064377:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5b9cdf94-0180-42f8-9a71-beb807b227b6:2489:1853163904548608:192.168.202.140@tcp:9:umount.0 00000100:00080000:0.0:1767316272.064546:0:46601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:0.0:1767316272.064551:0:46601:0:(client.c:3081:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9d1f90a81800: committing for last_committed 0 gen 2 00000100:00100000:0.0:1767316272.064557:0:46601:0:(import.c:321:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1767316272.064736:0:46601:0:(genops.c:1485:class_disconnect()) disconnect: cookie 0xa4df836b1649113a 00000020:00000080:0.0:1767316272.064748:0:46601:0:(genops.c:904:class_export_put()) final put ffff9d1f8af58000/LOV_OSC_UUID 00000020:01000000:0.0:1767316272.064832:0:46601:0:(obd_config.c:2361:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff9d1f90a81800 (flags='') 00000020:00000080:0.0:1767316272.064838:0:46601:0:(obd_config.c:1389:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1767316272.064855:0:46601:0:(obd_config.c:886:class_cleanup()) lustre-OST0000-osc-ffff9d1f90a81800: forcing exports to disconnect: 0/3 00000020:00080000:0.0:1767316272.064859:0:46601:0:(genops.c:1575:class_disconnect_exports()) OBD device 10 (ffff9d1f8b39ca88) has no exports 00000020:00000080:1.0:1767316272.064951:0:42605:0:(genops.c:863:class_export_destroy()) destroying export ffff9d1f8af58000/LOV_OSC_UUID for lustre-OST0000-osc-ffff9d1f90a81800 00000008:01000000:0.0:1767316272.065014:0:46601:0:(obd_class.h:663:obd_cleanup_client_import()) lustre-OST0000-osc-ffff9d1f90a81800: client import never connected 00000100:00080000:0.0:1767316272.065017:0:46601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:0.0:1767316272.065020:0:46601:0:(client.c:3081:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9d1f90a81800: committing for last_committed 0 gen 3 00000100:00020000:0.0:1767316272.065023:0:46601:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup 00000100:00100000:0.0:1767316272.065026:0:46601:0:(import.c:321:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000100:00020000:0.0:1767316272.065028:0:46601:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup 00000020:00000080:0.0:1767316272.065278:0:46601:0:(obd_config.c:1389:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1767316272.065286:0:46601:0:(obd_config.c:814:class_detach()) detach on obd lustre-OST0000-osc-ffff9d1f90a81800 (uuid 5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000020:00000080:1.0:1767316272.065450:0:42605:0:(genops.c:1202:obd_zombie_import_free()) destroying import ffff9d1f942fe800 for lustre-OST0000-osc-ffff9d1f90a81800 00000020:00000080:1.0:1767316272.065457:0:42605:0:(genops.c:904:class_export_put()) final put ffff9d1f942f9800/5b9cdf94-0180-42f8-9a71-beb807b227b6 00000020:00000080:1.0:1767316272.065469:0:42605:0:(genops.c:863:class_export_destroy()) destroying export ffff9d1f942f9800/5b9cdf94-0180-42f8-9a71-beb807b227b6 for lustre-OST0000-osc-ffff9d1f90a81800 00000020:01000000:1.0:1767316272.065475:0:42605:0:(genops.c:443:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff9d1f90a81800 (5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000020:00000080:0.0:1767316272.065724:0:46601:0:(genops.c:1485:class_disconnect()) disconnect: cookie 0xa4df836b16491133 00000020:00000080:0.0:1767316272.065730:0:46601:0:(genops.c:904:class_export_put()) final put ffff9d1f8af5c000/5b9cdf94-0180-42f8-9a71-beb807b227b6 00000100:00080000:0.0:1767316272.066041:0:46601:0:(pinger.c:396:ptlrpc_pinger_del_import()) removing pingable import 5b9cdf94-0180-42f8-9a71-beb807b227b6->lustre-MDT0000_UUID 00010000:00010000:0.0:1767316272.066159:0:46601:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.066207:0:46601:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316272.066215:0:46601:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1767316272.066230:0:46601:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a4d240 going to sleep for 0 seconds 00000100:00100000:0.0:1767316272.066239:0:46601:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job umount:5b9cdf94-0180-42f8-9a71-beb807b227b6:46601:1853163904548736:192.168.202.140@tcp:103:umount.0 00010000:00010000:0.0:1767316272.066248:0:46601:0:(ldlm_request.c:1494:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1767316272.066254:0:46601:0:(ldlm_request.c:2263:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1767316272.066257:0:46601:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f90a81800 lock: ffff9d1f93c85400/0xa4df836b16491179 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9806 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000020:00000080:1.0:1767316272.066524:0:42605:0:(genops.c:863:class_export_destroy()) destroying export ffff9d1f8af5c000/5b9cdf94-0180-42f8-9a71-beb807b227b6 for lustre-clilov-ffff9d1f90a81800 00000100:00080000:1.0:1767316272.070654:0:46601:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad2f000 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1767316272.070788:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5b9cdf94-0180-42f8-9a71-beb807b227b6:2489:1853163904548864:192.168.202.140@tcp:39:umount.0 00000100:00080000:2.0:1767316272.074256:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad2f000 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1767316272.074287:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5b9cdf94-0180-42f8-9a71-beb807b227b6:2489:1853163904548864:192.168.202.140@tcp:39:umount.0 00000100:00080000:1.0:1767316272.077322:0:46601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1767316272.077341:0:46601:0:(client.c:3081:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d1f90a81800: committing for last_committed 51539607558 gen 2 00000100:00100000:1.0:1767316272.077358:0:46601:0:(client.c:3119:ptlrpc_free_committed()) @@@ free request with old gen req@ffff9d1f918d0380 x1853163904548864/t51539607559(51539607559) o39->lustre-MDT0000-mdc-ffff9d1f90a81800@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 0 dl 1767316282 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316272.077389:0:46601:0:(import.c:321:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1767316272.079346:0:46601:0:(genops.c:1485:class_disconnect()) disconnect: cookie 0xa4df836b1649112c 00000020:00000080:1.0:1767316272.079361:0:46601:0:(genops.c:904:class_export_put()) final put ffff9d1f98817000/5b9cdf94-0180-42f8-9a71-beb807b227b6 00000020:00000080:1.0:1767316272.079409:0:46601:0:(genops.c:1485:class_disconnect()) disconnect: cookie 0xa4df836b16491125 00000020:00000080:1.0:1767316272.079413:0:46601:0:(genops.c:904:class_export_put()) final put ffff9d1f98814800/5b9cdf94-0180-42f8-9a71-beb807b227b6 00000020:01000000:1.0:1767316272.079423:0:46601:0:(obd_config.c:2361:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff9d1f90a81800 (flags='') 00000020:00000080:1.0:1767316272.079444:0:46601:0:(obd_config.c:1389:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1767316272.079452:0:46601:0:(obd_config.c:886:class_cleanup()) lustre-clilov-ffff9d1f90a81800: forcing exports to disconnect: 0/1 00000020:00080000:1.0:1767316272.079457:0:46601:0:(genops.c:1575:class_disconnect_exports()) OBD device 6 (ffff9d1f8b3998d8) has no exports 00000020:00000080:1.0:1767316272.079488:0:46601:0:(obd_config.c:1389:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1767316272.079493:0:46601:0:(obd_config.c:814:class_detach()) detach on obd lustre-clilov-ffff9d1f90a81800 (uuid 5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000020:00000080:1.0:1767316272.079496:0:46601:0:(genops.c:904:class_export_put()) final put ffff9d1f99a60800/5b9cdf94-0180-42f8-9a71-beb807b227b6 00000020:00000080:1.0:1767316272.079499:0:46601:0:(genops.c:863:class_export_destroy()) destroying export ffff9d1f99a60800/5b9cdf94-0180-42f8-9a71-beb807b227b6 for lustre-clilov-ffff9d1f90a81800 00000020:01000000:1.0:1767316272.079508:0:46601:0:(genops.c:443:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff9d1f90a81800 (5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000020:01000000:1.0:1767316272.079517:0:46601:0:(obd_config.c:2361:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff9d1f90a81800 (flags='') 00000020:00000080:1.0:1767316272.079521:0:46601:0:(obd_config.c:1389:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1767316272.079525:0:46601:0:(obd_config.c:886:class_cleanup()) lustre-clilmv-ffff9d1f90a81800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1767316272.079527:0:46601:0:(genops.c:1575:class_disconnect_exports()) OBD device 7 (ffff9d1f8b39e360) has no exports 00000020:00000080:3.0:1767316272.079535:0:42605:0:(genops.c:863:class_export_destroy()) destroying export ffff9d1f98817000/5b9cdf94-0180-42f8-9a71-beb807b227b6 for lustre-MDT0000-mdc-ffff9d1f90a81800 00000020:00000080:3.0:1767316272.079547:0:42605:0:(genops.c:863:class_export_destroy()) destroying export ffff9d1f98814800/5b9cdf94-0180-42f8-9a71-beb807b227b6 for lustre-clilmv-ffff9d1f90a81800 00000020:00000080:1.0:1767316272.079587:0:46601:0:(obd_config.c:1389:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1767316272.079592:0:46601:0:(obd_config.c:814:class_detach()) detach on obd lustre-clilmv-ffff9d1f90a81800 (uuid 5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000020:00000080:1.0:1767316272.079608:0:46601:0:(genops.c:904:class_export_put()) final put ffff9d1f99a64800/5b9cdf94-0180-42f8-9a71-beb807b227b6 00000020:00000080:1.0:1767316272.079610:0:46601:0:(genops.c:863:class_export_destroy()) destroying export ffff9d1f99a64800/5b9cdf94-0180-42f8-9a71-beb807b227b6 for lustre-clilmv-ffff9d1f90a81800 00000020:01000000:1.0:1767316272.079615:0:46601:0:(genops.c:443:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff9d1f90a81800 (5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000020:01000000:1.0:1767316272.079811:0:46601:0:(obd_config.c:2361:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff9d1f90a81800 (flags='') 00000020:00000080:1.0:1767316272.079815:0:46601:0:(obd_config.c:1389:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1767316272.079818:0:46601:0:(obd_config.c:886:class_cleanup()) lustre-MDT0000-mdc-ffff9d1f90a81800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1767316272.079820:0:46601:0:(genops.c:1575:class_disconnect_exports()) OBD device 8 (ffff9d1f8b398000) has no exports 00000008:01000000:1.0:1767316272.079841:0:46601:0:(obd_class.h:663:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff9d1f90a81800: client import never connected 00000100:00080000:1.0:1767316272.079846:0:46601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1767316272.079850:0:46601:0:(client.c:3081:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d1f90a81800: committing for last_committed 51539607558 gen 3 00000100:00020000:1.0:1767316272.079855:0:46601:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup 00000100:00100000:1.0:1767316272.079857:0:46601:0:(import.c:321:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000100:00020000:1.0:1767316272.079860:0:46601:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup 00000020:00000080:1.0:1767316272.080085:0:46601:0:(obd_config.c:1389:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1767316272.080090:0:46601:0:(obd_config.c:814:class_detach()) detach on obd lustre-MDT0000-mdc-ffff9d1f90a81800 (uuid 5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000080:02000400:1.0:1767316272.080094:0:46601:0:(llite_lib.c:1613:ll_put_super()) Unmounted lustre-client 00000020:00000080:0.0:1767316272.080140:0:36:0:(genops.c:1202:obd_zombie_import_free()) destroying import ffff9d1f8ad2f000 for lustre-MDT0000-mdc-ffff9d1f90a81800 00000020:00000080:0.0:1767316272.080146:0:36:0:(genops.c:904:class_export_put()) final put ffff9d1f8ad2a000/5b9cdf94-0180-42f8-9a71-beb807b227b6 00000020:00000080:0.0:1767316272.080148:0:36:0:(genops.c:863:class_export_destroy()) destroying export ffff9d1f8ad2a000/5b9cdf94-0180-42f8-9a71-beb807b227b6 for lustre-MDT0000-mdc-ffff9d1f90a81800 00000020:01000000:0.0:1767316272.080160:0:36:0:(genops.c:443:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff9d1f90a81800 (5b9cdf94-0180-42f8-9a71-beb807b227b6) 00000020:01000000:1.0:1767316272.096023:0:46601:0:(obd_config.c:1123:class_del_profile()) Del profile lustre-client 00000020:01000004:1.0:1767316272.096103:0:46601:0:(obd_mount.c:875:lustre_common_put_super()) dropping sb ffff9d1f90a81800 00000020:01000004:1.0:1767316272.096108:0:46601:0:(obd_mount.c:569:lustre_stop_mgc()) MGC still has 1 references. 00000020:01000004:1.0:1767316272.096110:0:46601:0:(obd_mount.c:888:lustre_common_put_super()) MGC still in use 00000020:01000004:1.0:1767316272.096112:0:46601:0:(obd_mount.c:705:lustre_put_lsi()) put ffff9d1f90a81800 1 00000020:01000004:1.0:1767316272.096115:0:46601:0:(obd_mount.c:637:lustre_free_lsi()) Freeing lsi ffff9d1f90a82000 00010000:00010000:3.0:1767316273.081208:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1767316273.655166:0:46733:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test complete, duration 1286 sec ======== 20:11:12 (1767316272) 00010000:00010000:2.0:1767316274.102778:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316275.129126:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1767316275.176176:0:46877:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: === replay-single: start cleanup 20:11:13 (1767316273) === 00000080:00200000:2.0:1767316275.497133:0:9368:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316275.497160:0:9368:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.497180:0:9368:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.497197:0:9368:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.497206:0:9368:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.497213:0:9368:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316275.497217:0:9368:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491172 00000080:00010000:2.0:1767316275.497225:0:9368:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491172 from it ffffb28141c1fbb0 00010000:00010000:2.0:1767316275.497229:0:9368:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.497238:0:9368:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.497256:0:9368:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 44 00000080:00200008:2.0:1767316275.497265:0:9368:0:(file.c:1001:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), flags 304000 00000080:00200008:2.0:1767316275.497283:0:9368:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), request_mask 2047, flags 0, foreign 0 00000002:00010000:2.0:1767316275.497299:0:9368:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: readdir flags 0 00010000:00010000:2.0:1767316275.497306:0:9368:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.497322:0:9368:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1767316275.498365:0:9368:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7480 pname:cluuid:pid:xid:nid:opc:job bash:45d767c3-7963-411a-be8f-bcbeb1c06075:9368:1853163904548992:192.168.202.140@tcp:37:bash.0 00000100:00100000:2.0:1767316275.498542:0:9368:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759e40 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.516205:0:9368:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7480 pname:cluuid:pid:xid:nid:opc:job bash:45d767c3-7963-411a-be8f-bcbeb1c06075:9368:1853163904548992:192.168.202.140@tcp:37:bash.0 00010000:00010000:0.0:1767316275.517757:0:9368:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.517775:0:9368:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316275.518300:0:9368:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316275.518405:0:9368:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316275.518421:0:9368:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316275.518433:0:9368:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316275.519256:0:9368:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1767316275.519264:0:9368:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316275.519268:0:9368:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491172 00000080:00010000:1.0:1767316275.519275:0:9368:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491172 from it ffffb28141c1fcd8 00010000:00010000:1.0:1767316275.519278:0:9368:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316275.519286:0:9368:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316275.519925:0:9368:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) pos/size0/10752 32bit_api 0 00000080:00200000:1.0:1767316275.519992:0:9368:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348) pos/size18446744073709551614/10752 32bit_api 0 00000080:00200008:1.0:1767316275.520295:0:9368:0:(file.c:424:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), flags 304000 00000080:00200000:1.0:1767316275.530609:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316275.530943:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316275.530963:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316275.530982:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316275.530990:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1767316275.531000:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:1.0:1767316275.531079:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491172 00000080:00010000:1.0:1767316275.531088:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491172 from it ffffb2814198fb58 00010000:00010000:1.0:1767316275.531092:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316275.531114:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316275.531130:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:1.0:1767316275.531144:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=d103.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=8 00000080:00200000:1.0:1767316275.531148:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=d103.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:1.0:1767316275.531162:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d103.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316275.531296:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:1.0:1767316275.531305:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:1.0:1767316275.531403:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:1.0:1767316275.531432:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904549120:192.168.202.140@tcp:101:rm.0 00000100:00100000:1.0:1767316275.531714:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f874c5180 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.547102:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904549120:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.547128:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.547153:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.547161:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.547171:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.547193:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.547205:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.547217:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.547245:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.547258:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f918d0380 x1853163904549120/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/584 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.547270:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.547281:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1034 (0 0) 00000080:00200000:2.0:1767316275.547288:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316275.547310:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x40a:0x0], updating i_size 11776 00000080:00200000:2.0:1767316275.547314:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x40a:0x0](ffff9d1f885d21c8): rc = 0 00000080:00010000:2.0:1767316275.547320:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40a:0x0](ffff9d1f885d21c8) for lock 0xa4df836b16491180 00000080:00010000:2.0:1767316275.547331:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40a:0x0](ffff9d1f885d21c8) 00000080:00010000:2.0:1767316275.547335:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491180 from it ffffb2814198fb20 00010000:00010000:2.0:1767316275.547338:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.547346:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200008:2.0:1767316275.547385:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d103.replay-single:[0x200000bd1:0x40a:0x0](ffff9d1f885d21c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316275.547393:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff9d1f885d21c8),name=d103.replay-single 00000002:00010000:2.0:1767316275.547425:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x40a:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.547433:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.547463:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.547470:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.547476:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40a:0x0](ffff9d1f885d21c8) 00000080:00010000:2.0:1767316275.547480:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40a:0x0](ffff9d1f885d21c8) for lock 0xa4df836b16491180 00000080:00010000:2.0:1767316275.547484:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491180 from it ffffb2814198fcb8 00010000:00010000:2.0:1767316275.547487:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.547493:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.547835:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316275.547845:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.547853:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.547862:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.547868:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.547872:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316275.547875:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491172 00000080:00010000:2.0:1767316275.547878:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491172 from it ffffb2814198fbf8 00010000:00010000:2.0:1767316275.547881:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.547898:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.547908:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.547915:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=266 00000080:00200000:2.0:1767316275.547922:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff9d1f885d21c8), inode mode 41ed mask 44 00000080:00200008:2.0:1767316275.547930:0:46893:0:(file.c:1001:ll_file_open()) START file d103.replay-single:[0x200000bd1:0x40a:0x0](ffff9d1f885d21c8), flags 704400 00000002:00010000:2.0:1767316275.547947:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x40a:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1767316275.547999:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b16491187 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.548006:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b16491187 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000020:00100000:2.0:1767316275.548018:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1767316275.548026:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4a80 x1853163904549376/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/133032 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1767316275.548035:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b16491187 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.548055:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904549376:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.548227:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967599c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.552224:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904549376:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.552249:0:46893:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b16491187 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.552262:0:46893:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b16491187 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.552287:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b16491187 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.552293:0:46893:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b16491187 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.552300:0:46893:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b16491187 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.552306:0:46893:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f909f3c00 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:2.0:1767316275.552312:0:46893:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b16491187 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.552319:0:46893:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b16491187 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.552326:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b16491187 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.552332:0:46893:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1767316275.552339:0:46893:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b16491187 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.552345:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b16491187 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00000002:00100000:2.0:1767316275.554239:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9d1f890e4a80 x1853163904549376/t51539607560(51539607560) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1767316275.554259:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4a80 x1853163904549376/t51539607560(51539607560) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1767316275.554272:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316275.554307:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x40a:0x0], updating i_size 11776 00000002:00100000:2.0:1767316275.554324:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4a80 x1853163904549376/t51539607560(51539607560) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1767316275.554441:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d103.replay-single:[0x200000bd1:0x40a:0x0](ffff9d1f885d21c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316275.554449:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff9d1f885d21c8),name=d103.replay-single 00000002:00010000:2.0:1767316275.554468:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x40a:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.554484:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.554498:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.554504:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.554510:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40a:0x0](ffff9d1f885d21c8) 00000080:00010000:2.0:1767316275.554513:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40a:0x0](ffff9d1f885d21c8) for lock 0xa4df836b16491180 00000080:00010000:2.0:1767316275.554519:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491180 from it ffffb2814198fcd8 00010000:00010000:2.0:1767316275.554522:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.554530:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.554601:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff9d1f885d21c8) pos/size0/11776 32bit_api 0 00000002:00010000:2.0:1767316275.554614:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x40a:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: readdir flags 0 00010000:00010000:2.0:1767316275.554620:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.554628:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1767316275.555378:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904549504:192.168.202.140@tcp:37:rm.0 00000100:00100000:2.0:1767316275.555553:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967599c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.568583:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904549504:192.168.202.140@tcp:37:rm.0 00010000:00010000:2.0:1767316275.569956:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.569977:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.570034:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff9d1f885d21c8) pos/size18446744073709551614/11776 32bit_api 0 00000080:00200008:2.0:1767316275.570124:0:46893:0:(file.c:424:ll_file_release()) START file d103.replay-single:[0x200000bd1:0x40a:0x0](ffff9d1f885d21c8), flags 704000 00000002:00100000:2.0:1767316275.570245:0:46893:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f890e4a80 x1853163904549376/t51539607560(51539607560) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1767316275.570258:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1767316275.570263:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e6300 x1853163904549760/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/1800 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1767316275.570287:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904549760:192.168.202.140@tcp:35:rm.0 00000100:00100000:2.0:1767316275.570472:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967599c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.575274:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904549760:192.168.202.140@tcp:35:rm.0 00000002:00100000:2.0:1767316275.575307:0:46893:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f890e4a80 x1853163904549376/t51539607560(51539607560) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1767316275.575375:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316275.575403:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.575438:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.575453:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.575461:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.575467:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316275.575471:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491172 00000080:00010000:2.0:1767316275.575477:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491172 from it ffffb2814198fc10 00010000:00010000:2.0:1767316275.575482:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.575491:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.575507:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.575519:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=0 00000080:00200000:2.0:1767316275.575525:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316275.575533:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.575538:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.575547:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.575553:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.575556:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316275.575559:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491172 00000080:00010000:2.0:1767316275.575563:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491172 from it ffffb2814198fd88 00010000:00010000:2.0:1767316275.575571:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.575579:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.575588:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 3 00000080:00200000:2.0:1767316275.575594:0:46893:0:(namei.c:2264:ll_rmdir()) VFS Op:name=d103.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348) 00010000:00010000:2.0:1767316275.575613:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.575755:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.575819:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.575826:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.575832:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1767316275.575834:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccce00/0xa4df836b16491180 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9822 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.575848:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd800/0xa4df836b16491172 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c97f8 expref: -99 pid: 46593 timeout: 0 lvb_type: 0 00000020:00100000:2.0:1767316275.575864:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316275.575871:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e6300 x1853163904550016/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 504/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316275.575912:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904550016:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316275.575990:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967599c0 going to sleep for 11 seconds 00000100:00100000:3.0:1767316275.587233:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904550016:192.168.202.140@tcp:36:rm.0 00000080:00200000:3.0:1767316275.587279:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff9d1f885d21c8) 00000080:00200000:3.0:1767316275.587355:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:3.0:1767316275.587373:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316275.587439:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316275.587450:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316275.587459:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:3.0:1767316275.587499:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904550144:192.168.202.140@tcp:101:rm.0 00000100:00100000:3.0:1767316275.587583:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f88c99d80 going to sleep for 11 seconds 00000100:00100000:3.0:1767316275.599921:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904550144:192.168.202.140@tcp:101:rm.0 00010000:00010000:3.0:1767316275.599965:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.599979:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.599987:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.599994:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.600002:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.600007:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.600013:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1767316275.600022:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f92f72680 x1853163904550144/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1767316275.600036:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.600065:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1767316275.600077:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1767316275.600100:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:3.0:1767316275.600105:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b1649118e 00000080:00010000:3.0:1767316275.600110:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:3.0:1767316275.600113:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649118e from it ffffb2814198fb58 00010000:00010000:3.0:1767316275.600116:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.600124:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316275.600147:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:3.0:1767316275.600169:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=8 00000080:00200000:3.0:1767316275.600174:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:3.0:1767316275.600196:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d130b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316275.600242:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316275.600250:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316275.600260:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:3.0:1767316275.600283:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904550272:192.168.202.140@tcp:101:rm.0 00000100:00100000:3.0:1767316275.600369:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f88c99d80 going to sleep for 11 seconds 00000100:00100000:3.0:1767316275.609982:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904550272:192.168.202.140@tcp:101:rm.0 00010000:00010000:3.0:1767316275.610033:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.610055:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.610063:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.610084:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.610096:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.610107:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.610114:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.610121:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1767316275.610139:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f92f72680 x1853163904550272/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/584 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1767316275.610152:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.610176:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1068 (0 0) 00000080:00200000:3.0:1767316275.610183:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1767316275.610207:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x42c:0x0], updating i_size 11776 00000080:00200000:3.0:1767316275.610210:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x42c:0x0](ffff9d1f885d3a88): rc = 0 00000080:00010000:3.0:1767316275.610217:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) for lock 0xa4df836b16491195 00000080:00010000:3.0:1767316275.610234:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) 00000080:00010000:3.0:1767316275.610237:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491195 from it ffffb2814198fb20 00010000:00010000:3.0:1767316275.610240:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.610248:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200008:3.0:1767316275.610288:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d130b.replay-single:[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1767316275.610295:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88),name=d130b.replay-single 00000002:00010000:3.0:1767316275.610314:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x42c:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316275.610322:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.610332:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316275.610338:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1767316275.610343:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) 00000080:00010000:3.0:1767316275.610346:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) for lock 0xa4df836b16491195 00000080:00010000:3.0:1767316275.610349:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491195 from it ffffb2814198fcb8 00010000:00010000:3.0:1767316275.610353:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.610366:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316275.610427:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:3.0:1767316275.610435:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316275.610442:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.610450:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316275.610456:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1767316275.610460:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:3.0:1767316275.610463:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b1649118e 00000080:00010000:3.0:1767316275.610466:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649118e from it ffffb2814198fbf8 00010000:00010000:3.0:1767316275.610469:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.610475:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316275.610483:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:3.0:1767316275.610490:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d130b.replay-single, flags=266 00000080:00200000:3.0:1767316275.610495:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88), inode mode 41ed mask 44 00000080:00200008:3.0:1767316275.610503:0:46893:0:(file.c:1001:ll_file_open()) START file d130b.replay-single:[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88), flags 704400 00000002:00010000:3.0:1767316275.610518:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x42c:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:3.0:1767316275.610563:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385600/0xa4df836b1649119c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316275.610575:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385600/0xa4df836b1649119c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000020:00100000:3.0:1767316275.610587:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1767316275.610594:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f92f72680 x1853163904550528/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/133032 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1767316275.610602:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385600/0xa4df836b1649119c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:3.0:1767316275.610622:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904550528:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.615123:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f88c99d80 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.619659:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904550528:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.619723:0:46893:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385600/0xa4df836b1649119c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.619736:0:46893:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385600/0xa4df836b1649119c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.619743:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385600/0xa4df836b1649119c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.619750:0:46893:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385600/0xa4df836b1649119c lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.619758:0:46893:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385600/0xa4df836b1649119c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.619764:0:46893:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f8b385600 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:2.0:1767316275.619769:0:46893:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385600/0xa4df836b1649119c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.619776:0:46893:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385600/0xa4df836b1649119c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.619783:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385600/0xa4df836b1649119c lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.619790:0:46893:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1767316275.619796:0:46893:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385600/0xa4df836b1649119c lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.619817:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385600/0xa4df836b1649119c lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00000002:00100000:2.0:1767316275.619838:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9d1f92f72680 x1853163904550528/t51539607563(51539607563) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1767316275.619851:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f92f72680 x1853163904550528/t51539607563(51539607563) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1767316275.619861:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316275.619880:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x42c:0x0], updating i_size 11776 00000002:00100000:2.0:1767316275.619892:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f92f72680 x1853163904550528/t51539607563(51539607563) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1767316275.619983:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d130b.replay-single:[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316275.619989:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88),name=d130b.replay-single 00000002:00010000:2.0:1767316275.620017:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x42c:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.620031:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.620042:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.620049:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.620054:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) 00000080:00010000:2.0:1767316275.620058:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) for lock 0xa4df836b16491195 00000080:00010000:2.0:1767316275.620063:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491195 from it ffffb2814198fcd8 00010000:00010000:2.0:1767316275.620066:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.620090:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.620350:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) pos/size0/11776 32bit_api 0 00000002:00010000:2.0:1767316275.620365:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x42c:0x0]) in obj [0x200000bd1:0x42c:0x0], intent: readdir flags 0 00010000:00010000:2.0:1767316275.620372:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.620381:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1767316275.620977:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904550656:192.168.202.140@tcp:37:rm.0 00000100:00100000:2.0:1767316275.621076:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759780 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.630230:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904550656:192.168.202.140@tcp:37:rm.0 00010000:00010000:2.0:1767316275.631220:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.631237:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200008:2.0:1767316275.631306:0:46893:0:(file.c:424:ll_file_release()) START file d130b.replay-single:[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88), flags 704000 00000002:00100000:2.0:1767316275.631372:0:46893:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f92f72680 x1853163904550528/t51539607563(51539607563) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1767316275.631384:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1767316275.631390:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4a80 x1853163904550912/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/1800 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1767316275.631443:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904550912:192.168.202.140@tcp:35:rm.0 00000100:00100000:2.0:1767316275.631536:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759780 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.635054:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904550912:192.168.202.140@tcp:35:rm.0 00000002:00100000:2.0:1767316275.635097:0:46893:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f92f72680 x1853163904550528/t51539607563(51539607563) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1767316275.635437:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316275.635458:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.635476:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.635490:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.635499:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.635504:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316275.635508:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b1649118e 00000080:00010000:2.0:1767316275.635516:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649118e from it ffffb2814198fbf8 00010000:00010000:2.0:1767316275.635519:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.635528:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.635546:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.635553:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d130b.replay-single, flags=266 00000080:00200000:2.0:1767316275.635561:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88), inode mode 41ed mask 44 00000080:00200008:2.0:1767316275.635588:0:46893:0:(file.c:1001:ll_file_open()) START file d130b.replay-single:[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88), flags 704400 00000002:00010000:2.0:1767316275.635605:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x42c:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:2.0:1767316275.635661:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.635669:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000020:00100000:2.0:1767316275.635791:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1767316275.635800:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4a80 x1853163904551168/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/133032 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1767316275.635810:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.635831:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904551168:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.636032:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759780 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.640601:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904551168:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.640643:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c988b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.640655:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c988b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.640667:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c988b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.640825:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c988b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.640843:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=200002b, status=0 req@ffff9d1f890e4a80 x1853163904551168/t51539607565(51539607565) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1767316275.640873:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4a80 x1853163904551168/t51539607565(51539607565) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.640884:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c988b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.640895:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 16 res 8589937617/1068 (0 0) 00000080:00200000:2.0:1767316275.640902:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316275.640924:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x42c:0x0], updating i_size 11776 00000080:00010000:2.0:1767316275.641001:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) for lock 0xa4df836b164911a3 00000080:00010000:2.0:1767316275.641010:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911a3 from it ffffb2814198fbd0 00010000:00010000:2.0:1767316275.641069:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c988b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.641078:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c988b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.641105:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4a80 x1853163904551168/t51539607565(51539607565) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1767316275.641188:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d130b.replay-single:[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316275.641195:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88),name=d130b.replay-single 00000002:00010000:2.0:1767316275.641215:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x42c:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.641225:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.641235:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.641254:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.641259:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) 00000080:00010000:2.0:1767316275.641263:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) for lock 0xa4df836b16491195 00000080:00010000:2.0:1767316275.641267:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491195 from it ffffb2814198fcd8 00010000:00010000:2.0:1767316275.641271:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.641278:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.641306:0:46893:0:(llite_lib.c:2766:ll_statfs()) VFS Op:sb=lustre (ffff9d1f98126000) d130b.replay-single proj=0 00000080:00000004:2.0:1767316275.641310:0:46893:0:(obd_class.h:1083:obd_statfs()) lustre-clilmv-ffff9d1f98126000: age 670, max_age 1522 00800000:00000004:2.0:1767316275.641316:0:46893:0:(obd_class.h:1083:obd_statfs()) lustre-MDT0000-mdc-ffff9d1f98126000: age 1501, max_age 1522 00000100:00100000:2.0:1767316275.641350:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904551296:192.168.202.140@tcp:41:rm.0 00000100:00100000:2.0:1767316275.641449:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.644541:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904551296:192.168.202.140@tcp:41:rm.0 00800000:00000004:2.0:1767316275.644571:0:46893:0:(obd_class.h:1109:obd_statfs()) lustre-MDT0000-mdc-ffff9d1f98126000: update ffff9d1f90e03520 cache blocks 1882112/1885696 objects 235392/235760 00000080:00000004:2.0:1767316275.644577:0:46893:0:(obd_class.h:1109:obd_statfs()) lustre-clilmv-ffff9d1f98126000: update ffff9d1f90e00370 cache blocks 1882112/1885696 objects 235392/235760 00000080:00000004:2.0:1767316275.644580:0:46893:0:(llite_lib.c:2629:ll_statfs_internal()) MDC blocks 1882112/1885696 objects 235392/235760 00000080:00200000:2.0:1767316275.644611:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) pos/size0/11776 32bit_api 0 00000080:00200000:2.0:1767316275.644640:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) pos/size18446744073709551614/11776 32bit_api 0 00000080:00200000:2.0:1767316275.644665:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.644763:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=f130b.replay-single, dir=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88), flags=0 00000080:00200000:2.0:1767316275.644769:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=f130b.replay-single, dir=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88), intent=getattr 00000002:00010000:2.0:1767316275.644796:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f130b.replay-single,[0x0:0x0:0x0]) in obj [0x200000bd1:0x42c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.644871:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.644881:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.644891:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.644911:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904551424:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.645120:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759780 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.652778:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904551424:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.652806:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.652830:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.652837:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.652848:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.652859:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.652871:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.652896:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.652903:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.652915:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e5180 x1853163904551424/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/800 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.652927:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 216 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316275.652938:0:46893:0:(mdc_locks.c:969:mdc_finish_enqueue()) ### DoM lock is returned by: getattr, size: 0 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000002:00010000:2.0:1767316275.652947:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.652958:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1070 (0 0) 00000080:00200000:2.0:1767316275.652965:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x42c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316275.653541:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x42e:0x0](ffff9d1f885863c8): rc = 0 00000080:00200000:2.0:1767316275.653557:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x42e:0x0]: layout version change: 4294967294 -> 2 00000080:00010000:2.0:1767316275.653562:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42e:0x0](ffff9d1f885863c8) for lock 0xa4df836b164911aa 00000080:00010000:2.0:1767316275.653748:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42e:0x0](ffff9d1f885863c8) 00000080:00010000:2.0:1767316275.653753:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911aa from it ffffb2814198fdf8 00010000:00010000:2.0:1767316275.653758:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.653783:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.653819:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88), inode mode 41ed mask 3 00000080:00200000:2.0:1767316275.653827:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=f130b.replay-single, dir=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) 00010000:00010000:2.0:1767316275.653860:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.653878:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c988b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.653950:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.653969:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x42e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1767316275.653981:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x42e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316275.653989:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x42e:0x0]: losing layout lock 00010000:00010000:2.0:1767316275.654493:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.654502:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.654508:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1767316275.654511:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911aa lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c98ae expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.654528:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385200/0xa4df836b16491195 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c986f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000020:00100000:2.0:1767316275.654553:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316275.654564:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e5180 x1853163904551680/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 504/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316275.654589:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904551680:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316275.654844:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759780 going to sleep for 11 seconds 00000100:00100000:3.0:1767316275.662798:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904551680:192.168.202.140@tcp:36:rm.0 00000080:00200000:3.0:1767316275.662883:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x42e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1767316275.662895:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:3.0:1767316275.662916:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:3.0:1767316275.662936:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x42e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1767316275.662943:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x42e:0x0](ffff9d1f885863c8) 00000080:00200008:3.0:1767316275.663019:0:46893:0:(file.c:424:ll_file_release()) START file d130b.replay-single:[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88), flags 704000 00010000:00010000:3.0:1767316275.663044:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c988b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316275.663102:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:3.0:1767316275.663131:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316275.663140:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.663149:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316275.664162:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1767316275.664185:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:3.0:1767316275.664189:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b1649118e 00000080:00010000:3.0:1767316275.664198:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649118e from it ffffb2814198fc10 00010000:00010000:3.0:1767316275.664203:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316275.664212:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1767316275.664226:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316275.664866:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=0 00000080:00200000:0.0:1767316275.664873:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:0.0:1767316275.664892:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d130b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316275.664943:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.664964:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.664975:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316275.665011:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904551808:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316275.665123:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a110c0 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.671392:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904551808:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316275.671427:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.671467:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.671476:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.671488:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.671500:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.671512:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.671520:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.671528:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316275.671540:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f91032300 x1853163904551808/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/584 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316275.671552:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.671562:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1068 (0 0) 00000080:00200000:0.0:1767316275.671569:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1767316275.671596:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x42c:0x0], updating i_size 11776 00000080:00200000:0.0:1767316275.671600:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x42c:0x0](ffff9d1f885d3a88): rc = 0 00000080:00010000:0.0:1767316275.671606:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) for lock 0xa4df836b164911b1 00000080:00010000:0.0:1767316275.671617:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) 00000080:00010000:0.0:1767316275.671621:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911b1 from it ffffb2814198fe00 00010000:00010000:0.0:1767316275.671624:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.671633:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316275.671662:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316275.671726:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316275.671736:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.671746:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316275.671752:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316275.671756:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316275.671759:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b1649118e 00000080:00010000:0.0:1767316275.671763:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649118e from it ffffb2814198fd88 00010000:00010000:0.0:1767316275.671766:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.671773:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316275.671782:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 3 00000080:00200000:0.0:1767316275.671796:0:46893:0:(namei.c:2264:ll_rmdir()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348) 00010000:00010000:0.0:1767316275.671815:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.671839:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.671849:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c988b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316275.671890:0:46893:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f890e4a80 x1853163904551168/t51539607565(51539607565) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1767316275.671902:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1767316275.671908:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91032300 x1853163904552064/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/1800 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1767316275.671929:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904552064:192.168.202.140@tcp:35:rm.0 00000100:00100000:0.0:1767316275.672018:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a110c0 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.677435:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904552064:192.168.202.140@tcp:35:rm.0 00000002:00100000:0.0:1767316275.677476:0:46893:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f890e4a80 x1853163904551168/t51539607565(51539607565) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1767316275.677563:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.677574:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c988b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.677581:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.677600:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1767316275.677603:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b1 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c98e6 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.677615:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3c00/0xa4df836b164911a3 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c988b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.677630:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385a00/0xa4df836b1649118e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9853 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000020:00100000:0.0:1767316275.677651:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1767316275.677658:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91032300 x1853163904552320/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 512/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767316275.677880:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904552320:192.168.202.140@tcp:36:rm.0 00000100:00100000:0.0:1767316275.678086:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a110c0 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.690368:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904552320:192.168.202.140@tcp:36:rm.0 00000080:00200000:0.0:1767316275.690419:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9d1f885d3a88) 00000080:00200000:0.0:1767316275.690491:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316275.690508:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316275.690558:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.690569:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.690579:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316275.690614:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904552448:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316275.690669:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a110c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.702509:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904552448:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.702534:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.702546:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.702557:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.702565:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.702575:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.702582:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.702589:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.702623:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f91030380 x1853163904552448/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.702637:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.702648:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1767316275.702655:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316275.702722:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1767316275.702730:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164911b8 00000080:00010000:2.0:1767316275.702736:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316275.702739:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911b8 from it ffffb2814198fb58 00010000:00010000:2.0:1767316275.702742:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.702750:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.702773:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.702804:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=8 00000080:00200000:2.0:1767316275.702809:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:2.0:1767316275.702828:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d134.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.702870:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.702876:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.702884:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.702910:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904552576:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.703019:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.710994:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904552576:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.711323:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.711419:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.711428:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.711439:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.711449:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.711461:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.711469:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.711477:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.711488:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e4e00 x1853163904552576/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/584 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.711516:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.711525:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1077 (0 0) 00000080:00200000:2.0:1767316275.711531:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316275.711553:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x435:0x0], updating i_size 11776 00000080:00200000:2.0:1767316275.711556:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x435:0x0](ffff9d1f885c5348): rc = 0 00000080:00010000:2.0:1767316275.711561:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x435:0x0](ffff9d1f885c5348) for lock 0xa4df836b164911bf 00000080:00010000:2.0:1767316275.711572:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x435:0x0](ffff9d1f885c5348) 00000080:00010000:2.0:1767316275.711576:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911bf from it ffffb2814198fb20 00010000:00010000:2.0:1767316275.711579:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.711588:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200008:2.0:1767316275.711637:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d134.replay-single:[0x200000bd1:0x435:0x0](ffff9d1f885c5348), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316275.711645:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348),name=d134.replay-single 00000002:00010000:2.0:1767316275.711665:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x435:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.711672:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.711709:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.711716:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.711722:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x435:0x0](ffff9d1f885c5348) 00000080:00010000:2.0:1767316275.711725:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x435:0x0](ffff9d1f885c5348) for lock 0xa4df836b164911bf 00000080:00010000:2.0:1767316275.711735:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911bf from it ffffb2814198fcb8 00010000:00010000:2.0:1767316275.711738:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.711746:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.711798:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316275.711804:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.711811:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.711820:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.711826:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.711830:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316275.711832:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164911b8 00000080:00010000:2.0:1767316275.711835:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911b8 from it ffffb2814198fbf8 00010000:00010000:2.0:1767316275.711839:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.711846:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.711853:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.711858:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d134.replay-single, flags=266 00000080:00200000:2.0:1767316275.711864:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), inode mode 41ed mask 44 00000080:00200008:2.0:1767316275.711872:0:46893:0:(file.c:1001:ll_file_open()) START file d134.replay-single:[0x200000bd1:0x435:0x0](ffff9d1f885c5348), flags 704400 00000002:00010000:2.0:1767316275.711890:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x435:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1767316275.711934:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911c6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.711941:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911c6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000020:00100000:2.0:1767316275.711951:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1767316275.711957:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4e00 x1853163904552832/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/133032 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1767316275.711966:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911c6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.711984:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904552832:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.712161:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.716544:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904552832:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.716565:0:46893:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911c6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.716578:0:46893:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911c6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.716586:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911c6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.716593:0:46893:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911c6 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.716600:0:46893:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911c6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.716628:0:46893:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f909f2c00 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:2.0:1767316275.716633:0:46893:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911c6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.716640:0:46893:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911c6 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.716647:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911c6 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.716654:0:46893:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1767316275.716660:0:46893:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911c6 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.716666:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911c6 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00000002:00100000:2.0:1767316275.716802:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9d1f890e4e00 x1853163904552832/t51539607569(51539607569) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1767316275.716815:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4e00 x1853163904552832/t51539607569(51539607569) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1767316275.716828:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316275.716847:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x435:0x0], updating i_size 11776 00000002:00100000:2.0:1767316275.716863:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4e00 x1853163904552832/t51539607569(51539607569) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1767316275.716955:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d134.replay-single:[0x200000bd1:0x435:0x0](ffff9d1f885c5348), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316275.716964:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348),name=d134.replay-single 00000002:00010000:2.0:1767316275.716983:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x435:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.717011:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.717023:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.717029:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.717035:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x435:0x0](ffff9d1f885c5348) 00000080:00010000:2.0:1767316275.717038:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x435:0x0](ffff9d1f885c5348) for lock 0xa4df836b164911bf 00000080:00010000:2.0:1767316275.717042:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911bf from it ffffb2814198fcd8 00010000:00010000:2.0:1767316275.717046:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.717052:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.717089:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348) pos/size0/11776 32bit_api 0 00000002:00010000:2.0:1767316275.717101:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x435:0x0]) in obj [0x200000bd1:0x435:0x0], intent: readdir flags 0 00010000:00010000:2.0:1767316275.717115:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.717122:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1767316275.719043:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904552960:192.168.202.140@tcp:37:rm.0 00000100:00100000:2.0:1767316275.719663:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.729755:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904552960:192.168.202.140@tcp:37:rm.0 00010000:00010000:2.0:1767316275.730804:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.730833:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200008:2.0:1767316275.730891:0:46893:0:(file.c:424:ll_file_release()) START file d134.replay-single:[0x200000bd1:0x435:0x0](ffff9d1f885c5348), flags 704000 00000002:00100000:2.0:1767316275.730951:0:46893:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f890e4e00 x1853163904552832/t51539607569(51539607569) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1767316275.730963:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1767316275.730968:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4700 x1853163904553216/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/1800 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1767316275.730992:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904553216:192.168.202.140@tcp:35:rm.0 00000100:00100000:2.0:1767316275.731084:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.734523:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904553216:192.168.202.140@tcp:35:rm.0 00000002:00100000:2.0:1767316275.734559:0:46893:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f890e4e00 x1853163904552832/t51539607569(51539607569) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1767316275.734634:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316275.734651:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.734669:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.734734:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.734743:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.734749:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316275.734753:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164911b8 00000080:00010000:2.0:1767316275.734773:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911b8 from it ffffb2814198fbf8 00010000:00010000:2.0:1767316275.734777:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.734786:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.734799:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.734807:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d134.replay-single, flags=266 00000080:00200000:2.0:1767316275.734814:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), inode mode 41ed mask 44 00000080:00200008:2.0:1767316275.734822:0:46893:0:(file.c:1001:ll_file_open()) START file d134.replay-single:[0x200000bd1:0x435:0x0](ffff9d1f885c5348), flags 704400 00000002:00010000:2.0:1767316275.734837:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x435:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:2.0:1767316275.734887:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.734895:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000020:00100000:2.0:1767316275.734906:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1767316275.734913:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4700 x1853163904553472/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/133032 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1767316275.734922:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.734942:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904553472:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.735601:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.740139:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904553472:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.740173:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9941 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.740203:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9941 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.740217:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9941 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.740226:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9941 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.740242:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=200002b, status=0 req@ffff9d1f890e4700 x1853163904553472/t51539607571(51539607571) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1767316275.740255:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4700 x1853163904553472/t51539607571(51539607571) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.740266:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9941 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.740275:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 16 res 8589937617/1077 (0 0) 00000080:00200000:2.0:1767316275.740282:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316275.740302:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x435:0x0], updating i_size 11776 00000080:00010000:2.0:1767316275.740308:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x435:0x0](ffff9d1f885c5348) for lock 0xa4df836b164911cd 00000080:00010000:2.0:1767316275.740316:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911cd from it ffffb2814198fbd0 00010000:00010000:2.0:1767316275.740320:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9941 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.740329:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9941 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.740376:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4700 x1853163904553472/t51539607571(51539607571) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1767316275.740469:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d134.replay-single:[0x200000bd1:0x435:0x0](ffff9d1f885c5348), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316275.740477:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348),name=d134.replay-single 00000002:00010000:2.0:1767316275.740497:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x435:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.740507:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.740518:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.740523:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.740528:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x435:0x0](ffff9d1f885c5348) 00000080:00010000:2.0:1767316275.740532:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x435:0x0](ffff9d1f885c5348) for lock 0xa4df836b164911bf 00000080:00010000:2.0:1767316275.740535:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911bf from it ffffb2814198fcd8 00010000:00010000:2.0:1767316275.740538:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.740544:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.740574:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348) pos/size0/11776 32bit_api 0 00000080:00200000:2.0:1767316275.740598:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348) pos/size18446744073709551614/11776 32bit_api 0 00000080:00200000:2.0:1767316275.740618:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.740633:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=A, dir=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), flags=0 00000080:00200000:2.0:1767316275.740637:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=A, dir=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), intent=getattr 00000002:00010000:2.0:1767316275.740655:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: A,[0x0:0x0:0x0]) in obj [0x200000bd1:0x435:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.740734:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.740741:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.740749:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.740768:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904553600:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.740848:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.748953:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904553600:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.748983:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.749013:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.749023:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.749034:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.749046:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.749058:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.749085:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.749092:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.749103:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e4e00 x1853163904553600/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/584 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.749130:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.749140:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1078 (0 0) 00000080:00200000:2.0:1767316275.749879:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x435:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1767316275.749994:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x436:0x0], updating i_size 11776 00000080:00200000:0.0:1767316275.749999:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x436:0x0](ffff9d1f885842c8): rc = 0 00000080:00010000:0.0:1767316275.750007:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x436:0x0](ffff9d1f885842c8) for lock 0xa4df836b164911d4 00000080:00010000:0.0:1767316275.750020:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x436:0x0](ffff9d1f885842c8) 00000080:00010000:0.0:1767316275.750025:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911d4 from it ffffb2814198fb20 00010000:00010000:0.0:1767316275.750029:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.750038:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200008:0.0:1767316275.750081:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file A:[0x200000bd1:0x436:0x0](ffff9d1f885842c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1767316275.751089:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x436:0x0](ffff9d1f885842c8),name=A 00000002:00010000:0.0:1767316275.751122:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x436:0x0]) in obj [0x200000bd1:0x435:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316275.751137:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.751150:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316275.751158:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x435:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316275.751164:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x436:0x0](ffff9d1f885842c8) 00000080:00010000:0.0:1767316275.751168:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x436:0x0](ffff9d1f885842c8) for lock 0xa4df836b164911d4 00010000:00010000:2.0:1767316275.751389:0:46894:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 46894 initiator: MDT0 00010000:00010000:2.0:1767316275.751407:0:46894:0:(ldlm_request.c:1177:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46894 initiator: MDT0 00000100:00100000:2.0:1767316275.751506:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904553728:192.168.202.140@tcp:63:ll_sa.0 00000080:00010000:0.0:1767316275.751651:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911d4 from it ffffb2814198fcb8 00010000:00010000:0.0:1767316275.751658:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.751711:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316275.751769:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316275.751775:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=A, flags=258 00000080:00200000:0.0:1767316275.751786:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x436:0x0](ffff9d1f885842c8), inode mode 41ed mask 44 00000080:00200008:0.0:1767316275.751794:0:46893:0:(file.c:1001:ll_file_open()) START file A:[0x200000bd1:0x436:0x0](ffff9d1f885842c8), flags 704400 00000002:00010000:0.0:1767316275.751809:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x436:0x0]) in obj [0x200000bd1:0x435:0x0], intent: open flags 0100240704401 00010000:00010000:0.0:1767316275.751843:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.751849:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000020:00100000:0.0:1767316275.751859:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1767316275.751867:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91031180 x1853163904553984/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/133032 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1767316275.751877:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316275.751891:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904553984:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316275.752013:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.756927:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904553984:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316275.756949:0:46893:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.756961:0:46893:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.756985:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.756992:0:46893:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e2 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.757000:0:46893:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.757006:0:46893:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f93c85400 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:0.0:1767316275.757010:0:46893:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.757016:0:46893:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.757023:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e2 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x20e400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.757030:0:46893:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1767316275.757036:0:46893:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e2 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.757042:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e2 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00000002:00100000:0.0:1767316275.757061:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9d1f91031180 x1853163904553984/t51539607572(51539607572) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1767316275.757088:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f91031180 x1853163904553984/t51539607572(51539607572) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1767316275.757100:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000bd1:0x435:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1767316275.757126:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x436:0x0], updating i_size 11776 00000002:00100000:0.0:1767316275.757143:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f91031180 x1853163904553984/t51539607572(51539607572) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1767316275.757227:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file A:[0x200000bd1:0x436:0x0](ffff9d1f885842c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1767316275.757234:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x436:0x0](ffff9d1f885842c8),name=A 00000002:00010000:0.0:1767316275.757254:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x436:0x0]) in obj [0x200000bd1:0x435:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316275.757269:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.757281:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316275.757288:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x435:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316275.757294:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x436:0x0](ffff9d1f885842c8) 00000080:00010000:0.0:1767316275.757298:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x436:0x0](ffff9d1f885842c8) for lock 0xa4df836b164911d4 00000080:00010000:0.0:1767316275.757302:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911d4 from it ffffb2814198fcd8 00010000:00010000:0.0:1767316275.757306:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.757313:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316275.757501:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x436:0x0](ffff9d1f885842c8) pos/size0/11776 32bit_api 0 00000002:00010000:0.0:1767316275.757515:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x436:0x0]) in obj [0x200000bd1:0x436:0x0], intent: readdir flags 0 00010000:00010000:0.0:1767316275.757522:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.757530:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1767316275.758275:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904554112:192.168.202.140@tcp:37:rm.0 00000100:00100000:0.0:1767316275.758967:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.769974:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904554112:192.168.202.140@tcp:37:rm.0 00010000:00010000:0.0:1767316275.770706:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.770720:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200008:0.0:1767316275.770778:0:46893:0:(file.c:424:ll_file_release()) START file A:[0x200000bd1:0x436:0x0](ffff9d1f885842c8), flags 704000 00000002:00100000:0.0:1767316275.770836:0:46893:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f91031180 x1853163904553984/t51539607572(51539607572) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1767316275.770846:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1767316275.770852:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91033100 x1853163904554368/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/1800 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1767316275.770873:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904554368:192.168.202.140@tcp:35:rm.0 00000100:00100000:0.0:1767316275.770955:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.773492:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904554368:192.168.202.140@tcp:35:rm.0 00000002:00100000:0.0:1767316275.773519:0:46893:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f91031180 x1853163904553984/t51539607572(51539607572) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1767316275.773586:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316275.773594:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=A, flags=258 00000080:00200000:0.0:1767316275.773601:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x436:0x0](ffff9d1f885842c8), inode mode 41ed mask 44 00000080:00200008:0.0:1767316275.773609:0:46893:0:(file.c:1001:ll_file_open()) START file A:[0x200000bd1:0x436:0x0](ffff9d1f885842c8), flags 704400 00000002:00010000:0.0:1767316275.773644:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x436:0x0]) in obj [0x200000bd1:0x435:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1767316275.773827:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.773837:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000020:00100000:0.0:1767316275.773848:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1767316275.773854:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91033100 x1853163904554624/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/133032 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1767316275.773863:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316275.773880:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904554624:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316275.774032:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00010000:00010000:2.0:1767316275.774111:0:2492:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.774129:0:2492:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.774137:0:2492:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.774158:0:2492:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.774169:0:2492:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.774178:0:2492:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.774186:0:2492:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.774194:0:2492:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.774203:0:2492:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e5c00 x1853163904553728/t0(0) o63->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:24/4 lens 664/624 e 0 to 0 dl 1767316291 ref 1 fl Interpret:RQU/200/0 rc 0/0 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000080:00010000:2.0:1767316275.774214:0:2492:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911db from it ffff9d1f90e0a988 00010000:00010000:2.0:1767316275.774217:0:2492:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.774224:0:2492:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1767316275.774247:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904553728:192.168.202.140@tcp:63:ll_sa.0 00010000:00010000:2.0:1767316275.774274:0:39242:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.774303:0:39242:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x438:0x0], updating i_size 11776 00000080:00200000:2.0:1767316275.774307:0:39242:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x438:0x0](ffff9d1f88585b88): rc = 0 00000080:00010000:2.0:1767316275.774312:0:39242:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x438:0x0](ffff9d1f88585b88) for lock 0xa4df836b164911db 00000080:00010000:2.0:1767316275.774317:0:39242:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911db from it ffff9d1f90e0a988 00010000:00010000:2.0:1767316275.774320:0:39242:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.774328:0:39242:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1767316275.776876:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904554624:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316275.776907:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c999c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.776917:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c999c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.776928:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c999c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.776938:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c999c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316275.776951:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=200002b, status=0 req@ffff9d1f91033100 x1853163904554624/t51539607574(51539607574) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1767316275.776981:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f91033100 x1853163904554624/t51539607574(51539607574) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316275.776991:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c999c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.777001:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 16 res 8589937617/1078 (0 0) 00000080:00200000:0.0:1767316275.777008:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000bd1:0x435:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1767316275.777025:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x436:0x0], updating i_size 11776 00000080:00010000:0.0:1767316275.777030:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x436:0x0](ffff9d1f885842c8) for lock 0xa4df836b164911e9 00000080:00010000:0.0:1767316275.777037:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911e9 from it ffffb2814198fbd0 00010000:00010000:0.0:1767316275.777041:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c999c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.777049:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c999c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316275.777075:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f91033100 x1853163904554624/t51539607574(51539607574) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1767316275.777149:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file A:[0x200000bd1:0x436:0x0](ffff9d1f885842c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1767316275.777156:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x436:0x0](ffff9d1f885842c8),name=A 00000002:00010000:0.0:1767316275.777176:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x436:0x0]) in obj [0x200000bd1:0x435:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316275.777185:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.777196:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316275.777211:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x435:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316275.777216:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x436:0x0](ffff9d1f885842c8) 00000080:00010000:0.0:1767316275.777219:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x436:0x0](ffff9d1f885842c8) for lock 0xa4df836b164911d4 00000080:00010000:0.0:1767316275.777223:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911d4 from it ffffb2814198fcd8 00010000:00010000:0.0:1767316275.777226:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.777233:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316275.777264:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x436:0x0](ffff9d1f885842c8) pos/size0/11776 32bit_api 0 00000080:00200000:0.0:1767316275.777288:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x436:0x0](ffff9d1f885842c8) pos/size18446744073709551614/11776 32bit_api 0 00000080:00200000:0.0:1767316275.777309:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x436:0x0](ffff9d1f885842c8), inode mode 41ed mask 1 00000080:00200000:0.0:1767316275.777324:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=f134.replay-single, dir=[0x200000bd1:0x436:0x0](ffff9d1f885842c8), flags=0 00000080:00200000:0.0:1767316275.777328:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=f134.replay-single, dir=[0x200000bd1:0x436:0x0](ffff9d1f885842c8), intent=getattr 00000002:00010000:0.0:1767316275.777626:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f134.replay-single,[0x0:0x0:0x0]) in obj [0x200000bd1:0x436:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316275.777667:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.777673:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.777714:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316275.777733:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904554752:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316275.777810:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.784359:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904554752:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316275.784420:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.784448:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.784458:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.784473:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.784486:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.784499:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.784507:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.784515:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316275.784528:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f91031180 x1853163904554752/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/656 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316275.784541:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 72 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1767316275.784567:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316275.784578:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1082 (0 0) 00000080:00200000:0.0:1767316275.784587:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x436:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1767316275.784845:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x43a:0x0](ffff9d1f885c4b08): rc = 0 00000080:00200000:0.0:1767316275.784862:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x43a:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1767316275.784869:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x43a:0x0](ffff9d1f885c4b08) for lock 0xa4df836b164911f0 00000080:00010000:0.0:1767316275.784880:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x43a:0x0](ffff9d1f885c4b08) 00000080:00010000:0.0:1767316275.784884:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911f0 from it ffffb2814198fdf8 00010000:00010000:0.0:1767316275.784887:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316275.784895:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316275.784928:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x436:0x0](ffff9d1f885842c8), inode mode 41ed mask 3 00000080:00200000:0.0:1767316275.784936:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=f134.replay-single, dir=[0x200000bd1:0x436:0x0](ffff9d1f885842c8) 00010000:00010000:0.0:1767316275.784965:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.784984:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c999c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.785057:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316275.785161:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x43a:0x0]: losing layout lock 00010000:00010000:0.0:1767316275.785291:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316275.785298:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.785302:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1767316275.785305:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911f0 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x43a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c99bf expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316275.785322:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2000/0xa4df836b164911d4 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9964 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000020:00100000:0.0:1767316275.785413:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1767316275.785422:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91031180 x1853163904555008/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 504/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767316275.785449:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904555008:192.168.202.140@tcp:36:rm.0 00000100:00100000:0.0:1767316275.785616:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.791947:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904555008:192.168.202.140@tcp:36:rm.0 00000080:00200000:0.0:1767316275.792171:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x43a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1767316275.792482:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1767316275.792569:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x43a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1767316275.792579:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x43a:0x0](ffff9d1f885c4b08) 00000080:00200008:0.0:1767316275.792787:0:46893:0:(file.c:424:ll_file_release()) START file A:[0x200000bd1:0x436:0x0](ffff9d1f885842c8), flags 704000 00010000:00010000:0.0:1767316275.792814:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c999c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316275.792852:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316275.792882:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=A, dir=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), flags=0 00000080:00200000:0.0:1767316275.792887:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=A, dir=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), intent=getattr 00000002:00010000:0.0:1767316275.793156:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: A,[0x0:0x0:0x0]) in obj [0x200000bd1:0x435:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316275.793454:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.793463:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.793474:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316275.793501:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904555136:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316275.793650:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.799935:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904555136:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316275.799966:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.799987:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.799995:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.800810:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.800825:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.800869:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.800878:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.800885:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316275.800900:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f91030000 x1853163904555136/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/584 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316275.800913:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.800925:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1078 (0 0) 00000080:00200000:0.0:1767316275.800934:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x435:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1767316275.800960:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x436:0x0], updating i_size 11776 00000080:00200000:0.0:1767316275.800964:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x436:0x0](ffff9d1f885842c8): rc = 0 00000080:00010000:0.0:1767316275.800970:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x436:0x0](ffff9d1f885842c8) for lock 0xa4df836b164911f7 00000080:00010000:0.0:1767316275.800982:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x436:0x0](ffff9d1f885842c8) 00000080:00010000:0.0:1767316275.800986:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911f7 from it ffffb2814198fe00 00010000:00010000:0.0:1767316275.800989:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.800996:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316275.801487:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), inode mode 41ed mask 3 00000080:00200000:0.0:1767316275.801498:0:46893:0:(namei.c:2264:ll_rmdir()) VFS Op:name=A, dir=[0x200000bd1:0x435:0x0](ffff9d1f885c5348) 00010000:00010000:0.0:1767316275.801528:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.801546:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9941 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.801565:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.801574:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c999c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316275.801616:0:46893:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f91033100 x1853163904554624/t51539607574(51539607574) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1767316275.801629:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1767316275.801635:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91030000 x1853163904555392/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/1800 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1767316275.801663:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904555392:192.168.202.140@tcp:35:rm.0 00000100:00100000:0.0:1767316275.803724:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.807573:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904555392:192.168.202.140@tcp:35:rm.0 00000002:00100000:0.0:1767316275.807607:0:46893:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f91033100 x1853163904554624/t51539607574(51539607574) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1767316275.807709:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.807736:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c999c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.807743:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.807749:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1767316275.807752:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b164911f7 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x436:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c99f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.807764:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b164911e9 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x436:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c999c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.807776:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2800/0xa4df836b164911bf lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9925 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000020:00100000:0.0:1767316275.807793:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1767316275.807800:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91030000 x1853163904555648/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 496/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767316275.807823:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904555648:192.168.202.140@tcp:36:rm.0 00000100:00100000:0.0:1767316275.807940:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.812243:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904555648:192.168.202.140@tcp:36:rm.0 00000080:00200000:0.0:1767316275.812288:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x436:0x0](ffff9d1f885842c8) 00000080:00200000:0.0:1767316275.812359:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316275.812376:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=B, dir=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), flags=0 00000080:00200000:0.0:1767316275.812381:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=B, dir=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), intent=getattr 00010000:00010000:0.0:1767316275.812431:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000080:00010000:0.0:1767316275.812463:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911db from it ffffb2814198f910 00010000:00010000:0.0:1767316275.812469:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.812477:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000080:00200008:0.0:1767316275.812650:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file B:[0x200000bd1:0x438:0x0](ffff9d1f88585b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1767316275.812656:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x438:0x0](ffff9d1f88585b88),name=B 00000002:00010000:0.0:1767316275.812707:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x438:0x0]) in obj [0x200000bd1:0x435:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316275.812719:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.812730:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316275.812738:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x435:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316275.812745:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x438:0x0](ffff9d1f88585b88) 00000080:00010000:0.0:1767316275.812748:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x438:0x0](ffff9d1f88585b88) for lock 0xa4df836b164911db 00000080:00010000:0.0:1767316275.812754:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911db from it ffffb2814198fcb8 00010000:00010000:0.0:1767316275.812757:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.812764:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316275.812802:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316275.812816:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=B, flags=258 00000080:00200000:0.0:1767316275.812821:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x438:0x0](ffff9d1f88585b88), inode mode 41ed mask 44 00000080:00200008:0.0:1767316275.812829:0:46893:0:(file.c:1001:ll_file_open()) START file B:[0x200000bd1:0x438:0x0](ffff9d1f88585b88), flags 704400 00000002:00010000:0.0:1767316275.812845:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x438:0x0]) in obj [0x200000bd1:0x435:0x0], intent: open flags 0100240704401 00010000:00010000:0.0:1767316275.812898:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911fe lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.812906:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911fe lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000020:00100000:0.0:1767316275.812918:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1767316275.812927:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91033100 x1853163904555904/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/133032 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1767316275.812938:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911fe lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316275.812958:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904555904:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316275.813138:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.820795:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904555904:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316275.820820:0:46893:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911fe lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.820833:0:46893:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911fe lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.820840:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911fe lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.820847:0:46893:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911fe lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.820874:0:46893:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911fe lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.820881:0:46893:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f93c84c00 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:0.0:1767316275.820886:0:46893:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911fe lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.820894:0:46893:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911fe lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.820901:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911fe lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.820907:0:46893:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1767316275.820913:0:46893:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911fe lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316275.820919:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84c00/0xa4df836b164911fe lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00000002:00100000:0.0:1767316275.820937:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9d1f91033100 x1853163904555904/t51539607578(51539607578) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1767316275.820951:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f91033100 x1853163904555904/t51539607578(51539607578) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1767316275.820968:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000bd1:0x435:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1767316275.820990:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x438:0x0], updating i_size 11776 00000002:00100000:1.0:1767316275.821288:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f91033100 x1853163904555904/t51539607578(51539607578) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1767316275.821457:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file B:[0x200000bd1:0x438:0x0](ffff9d1f88585b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1767316275.821466:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x438:0x0](ffff9d1f88585b88),name=B 00000002:00010000:1.0:1767316275.821492:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x438:0x0]) in obj [0x200000bd1:0x435:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316275.821508:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316275.821519:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316275.822998:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x435:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1767316275.823012:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x438:0x0](ffff9d1f88585b88) 00000080:00010000:1.0:1767316275.823016:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x438:0x0](ffff9d1f88585b88) for lock 0xa4df836b164911db 00000080:00010000:1.0:1767316275.823025:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911db from it ffffb2814198fcd8 00010000:00010000:1.0:1767316275.823031:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316275.823042:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1767316275.823296:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x438:0x0](ffff9d1f88585b88) pos/size0/11776 32bit_api 0 00000002:00010000:1.0:1767316275.823329:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x438:0x0]) in obj [0x200000bd1:0x438:0x0], intent: readdir flags 0 00010000:00010000:1.0:1767316275.823727:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316275.823740:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000100:00100000:1.0:1767316275.824986:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904556032:192.168.202.140@tcp:37:rm.0 00000100:00100000:1.0:1767316275.825747:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f874c5840 going to sleep for 11 seconds 00000100:00100000:0.0:1767316275.837927:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904556032:192.168.202.140@tcp:37:rm.0 00010000:00010000:0.0:1767316275.838941:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316275.838978:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000080:00200008:2.0:1767316275.840450:0:46893:0:(file.c:424:ll_file_release()) START file B:[0x200000bd1:0x438:0x0](ffff9d1f88585b88), flags 704000 00000002:00100000:2.0:1767316275.840541:0:46893:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f91033100 x1853163904555904/t51539607578(51539607578) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1767316275.840555:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1767316275.840560:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e5c00 x1853163904556288/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/1800 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1767316275.840586:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904556288:192.168.202.140@tcp:35:rm.0 00000100:00100000:2.0:1767316275.840869:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.844359:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904556288:192.168.202.140@tcp:35:rm.0 00000002:00100000:2.0:1767316275.844404:0:46893:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f91033100 x1853163904555904/t51539607578(51539607578) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1767316275.844481:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.844491:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=B, flags=258 00000080:00200000:2.0:1767316275.844500:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x438:0x0](ffff9d1f88585b88), inode mode 41ed mask 44 00000080:00200008:2.0:1767316275.844508:0:46893:0:(file.c:1001:ll_file_open()) START file B:[0x200000bd1:0x438:0x0](ffff9d1f88585b88), flags 704400 00000002:00010000:2.0:1767316275.844550:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x438:0x0]) in obj [0x200000bd1:0x435:0x0], intent: open flags 0104240704401 00010000:00010000:2.0:1767316275.844623:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.844632:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000020:00100000:2.0:1767316275.844642:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1767316275.844649:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e5c00 x1853163904556544/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/133032 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1767316275.844657:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.844730:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904556544:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.844911:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.849477:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904556544:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.849514:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a28 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.849527:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a28 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.849541:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9a28 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.849555:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a28 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.849570:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=200002b, status=0 req@ffff9d1f890e5c00 x1853163904556544/t51539607580(51539607580) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1767316275.849605:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e5c00 x1853163904556544/t51539607580(51539607580) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.849618:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a28 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.849630:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 16 res 8589937617/1080 (0 0) 00000080:00200000:2.0:1767316275.849637:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000bd1:0x435:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316275.849659:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x438:0x0], updating i_size 11776 00000080:00010000:2.0:1767316275.849665:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x438:0x0](ffff9d1f88585b88) for lock 0xa4df836b16491205 00000080:00010000:2.0:1767316275.849675:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491205 from it ffffb2814198fbd0 00010000:00010000:2.0:1767316275.849702:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9a28 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.849712:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a28 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.849743:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e5c00 x1853163904556544/t51539607580(51539607580) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1767316275.849827:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file B:[0x200000bd1:0x438:0x0](ffff9d1f88585b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316275.849834:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x438:0x0](ffff9d1f88585b88),name=B 00000002:00010000:2.0:1767316275.849856:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x438:0x0]) in obj [0x200000bd1:0x435:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.849864:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.849871:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.849888:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x435:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.849894:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x438:0x0](ffff9d1f88585b88) 00000080:00010000:2.0:1767316275.849898:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x438:0x0](ffff9d1f88585b88) for lock 0xa4df836b164911db 00000080:00010000:2.0:1767316275.849904:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911db from it ffffb2814198fcd8 00010000:00010000:2.0:1767316275.849907:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.849913:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.849951:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x438:0x0](ffff9d1f88585b88) pos/size0/11776 32bit_api 0 00000080:00200000:2.0:1767316275.849974:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x438:0x0](ffff9d1f88585b88) pos/size18446744073709551614/11776 32bit_api 0 00000080:00200000:2.0:1767316275.849995:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x438:0x0](ffff9d1f88585b88), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.850023:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=f134.replay-single, dir=[0x200000bd1:0x438:0x0](ffff9d1f88585b88), flags=0 00000080:00200000:2.0:1767316275.850027:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=f134.replay-single, dir=[0x200000bd1:0x438:0x0](ffff9d1f88585b88), intent=getattr 00000002:00010000:2.0:1767316275.850038:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f134.replay-single,[0x0:0x0:0x0]) in obj [0x200000bd1:0x438:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.850082:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.850089:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.850098:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.850117:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6680 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904556672:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.850204:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.854537:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6680 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904556672:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.854586:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.854608:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.854617:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.854629:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.854639:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.854651:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.854659:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.854667:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.854728:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e6680 x1853163904556672/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/736 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.854740:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 152 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316275.854763:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.854774:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1083 (0 0) 00000080:00200000:2.0:1767316275.854781:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x438:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316275.854823:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x43b:0x0](ffff9d1f88582a08): rc = 0 00000080:00200000:2.0:1767316275.854836:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x43b:0x0]: layout version change: 4294967294 -> 1 00000080:00010000:2.0:1767316275.854841:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x43b:0x0](ffff9d1f88582a08) for lock 0xa4df836b1649120c 00000080:00010000:2.0:1767316275.854850:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x43b:0x0](ffff9d1f88582a08) 00000080:00010000:2.0:1767316275.854854:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649120c from it ffffb2814198fdf8 00010000:00010000:2.0:1767316275.854857:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.854865:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.854897:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x438:0x0](ffff9d1f88585b88), inode mode 41ed mask 3 00000080:00200000:2.0:1767316275.854906:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=f134.replay-single, dir=[0x200000bd1:0x438:0x0](ffff9d1f88585b88) 00010000:00010000:2.0:1767316275.854932:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.854949:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a28 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.855036:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.855047:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x43b:0x0]: losing layout lock 00010000:00010000:2.0:1767316275.855092:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.855099:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.855189:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1767316275.855193:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649120c lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x43b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9a4b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.855208:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164911db lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c998e expref: -99 pid: 46894 timeout: 0 lvb_type: 0 00000020:00100000:2.0:1767316275.855224:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316275.855232:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e6680 x1853163904556928/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 504/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316275.855254:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6680 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904556928:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316275.855409:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.866456:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6680 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904556928:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316275.866527:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x43b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316275.866542:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316275.866604:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x43b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316275.866612:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x43b:0x0](ffff9d1f88582a08) 00000080:00200008:2.0:1767316275.866836:0:46893:0:(file.c:424:ll_file_release()) START file B:[0x200000bd1:0x438:0x0](ffff9d1f88585b88), flags 704000 00010000:00010000:2.0:1767316275.866861:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a28 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.866900:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.866930:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=B, dir=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), flags=0 00000080:00200000:2.0:1767316275.866935:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=B, dir=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), intent=getattr 00000002:00010000:2.0:1767316275.866976:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: B,[0x0:0x0:0x0]) in obj [0x200000bd1:0x435:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.867044:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.867053:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.867063:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.867098:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904557056:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.867211:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.871715:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904557056:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.871747:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.871766:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.871775:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.871787:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.871798:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.871843:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.871852:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.871858:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.871870:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e4000 x1853163904557056/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/584 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.871884:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.871895:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1080 (0 0) 00000080:00200000:2.0:1767316275.871903:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x435:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316275.871928:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x438:0x0], updating i_size 11776 00000080:00200000:2.0:1767316275.871931:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x438:0x0](ffff9d1f88585b88): rc = 0 00000080:00010000:2.0:1767316275.871937:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x438:0x0](ffff9d1f88585b88) for lock 0xa4df836b16491213 00000080:00010000:2.0:1767316275.871949:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x438:0x0](ffff9d1f88585b88) 00000080:00010000:2.0:1767316275.871953:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491213 from it ffffb2814198fe00 00010000:00010000:2.0:1767316275.871956:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.871965:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.872024:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348), inode mode 41ed mask 3 00000080:00200000:2.0:1767316275.872035:0:46893:0:(namei.c:2264:ll_rmdir()) VFS Op:name=B, dir=[0x200000bd1:0x435:0x0](ffff9d1f885c5348) 00010000:00010000:2.0:1767316275.872065:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.872100:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9a28 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.872153:0:46893:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f890e5c00 x1853163904556544/t51539607580(51539607580) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1767316275.872165:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1767316275.872170:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4000 x1853163904557312/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/1800 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1767316275.872206:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904557312:192.168.202.140@tcp:35:rm.0 00000100:00100000:2.0:1767316275.872288:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.875842:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904557312:192.168.202.140@tcp:35:rm.0 00000002:00100000:2.0:1767316275.875885:0:46893:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f890e5c00 x1853163904556544/t51539607580(51539607580) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1767316275.875970:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.875981:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9a28 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.875987:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1767316275.876018:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491213 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x438:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9a83 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.876032:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491205 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x438:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9a28 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000020:00100000:2.0:1767316275.876056:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316275.876063:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4000 x1853163904557568/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316275.876097:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904557568:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316275.876195:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.880925:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904557568:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316275.880979:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x438:0x0](ffff9d1f88585b88) 00000080:00200008:2.0:1767316275.881054:0:46893:0:(file.c:424:ll_file_release()) START file d134.replay-single:[0x200000bd1:0x435:0x0](ffff9d1f885c5348), flags 704000 00010000:00010000:2.0:1767316275.881095:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9941 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.881142:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316275.881160:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.881169:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.881178:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.881185:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.881191:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316275.881194:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164911b8 00000080:00010000:2.0:1767316275.881220:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911b8 from it ffffb2814198fc10 00010000:00010000:2.0:1767316275.881224:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.881233:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.881248:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.881268:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=0 00000080:00200000:2.0:1767316275.881273:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:2.0:1767316275.881286:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d134.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.881331:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.881339:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.881349:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.881370:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904557696:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.881449:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.886119:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904557696:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.886152:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.886189:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.886198:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.886230:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.886242:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.886255:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.886263:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.886271:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.886283:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e5c00 x1853163904557696/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/584 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.886297:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.886308:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1077 (0 0) 00000080:00200000:2.0:1767316275.886315:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316275.886336:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x435:0x0], updating i_size 11776 00000080:00200000:2.0:1767316275.886339:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x435:0x0](ffff9d1f885c5348): rc = 0 00000080:00010000:2.0:1767316275.886345:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x435:0x0](ffff9d1f885c5348) for lock 0xa4df836b1649121a 00000080:00010000:2.0:1767316275.886356:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x435:0x0](ffff9d1f885c5348) 00000080:00010000:2.0:1767316275.886359:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649121a from it ffffb2814198fe00 00010000:00010000:2.0:1767316275.886368:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.886377:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.886421:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316275.886441:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.886450:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.886460:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.886466:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.886471:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316275.886474:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164911b8 00000080:00010000:2.0:1767316275.886478:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164911b8 from it ffffb2814198fd88 00010000:00010000:2.0:1767316275.886481:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.886488:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.886496:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 3 00000080:00200000:2.0:1767316275.886504:0:46893:0:(namei.c:2264:ll_rmdir()) VFS Op:name=d134.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348) 00010000:00010000:2.0:1767316275.886525:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.886549:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.886570:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9941 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.886613:0:46893:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f890e4700 x1853163904553472/t51539607571(51539607571) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1767316275.886625:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1767316275.886631:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e5c00 x1853163904557952/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/1800 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1767316275.886653:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904557952:192.168.202.140@tcp:35:rm.0 00000100:00100000:2.0:1767316275.886802:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.890341:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904557952:192.168.202.140@tcp:35:rm.0 00000002:00100000:2.0:1767316275.890374:0:46893:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f890e4700 x1853163904553472/t51539607571(51539607571) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1767316275.890474:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.890485:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9941 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.890492:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.890499:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:2.0:1767316275.890502:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649121a lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x435:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9ab4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.890540:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b164911cd lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x435:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9941 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.890554:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b164911b8 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9909 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000020:00100000:2.0:1767316275.890574:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316275.890580:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e5c00 x1853163904558208/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 512/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316275.890602:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904558208:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316275.890723:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.898292:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5c00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904558208:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316275.898334:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x435:0x0](ffff9d1f885c5348) 00000080:00200000:2.0:1767316275.898414:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316275.898433:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.898490:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.898501:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.898511:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.898531:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904558336:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.898623:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.902301:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904558336:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.902340:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.902353:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.902364:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.902375:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.902386:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.902402:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.902410:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.902421:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e4700 x1853163904558336/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.902435:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.902447:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1767316275.902454:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316275.902481:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1767316275.902494:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491221 00000080:00010000:2.0:1767316275.902500:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316275.902503:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491221 from it ffffb2814198fb58 00010000:00010000:2.0:1767316275.902506:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.902514:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.902540:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.902561:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=8 00000080:00200000:2.0:1767316275.902565:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:2.0:1767316275.902584:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d135.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.902622:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.902629:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.902637:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.902654:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904558464:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.902849:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.912383:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904558464:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.912441:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.912464:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.912500:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.912511:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.912522:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.912535:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.912543:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.912550:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.912562:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e4700 x1853163904558464/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/584 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.912574:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.912584:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1084 (0 0) 00000080:00200000:2.0:1767316275.912591:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316275.912613:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x43c:0x0], updating i_size 12800 00000080:00200000:2.0:1767316275.912617:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x43c:0x0](ffff9d1f885d1988): rc = 0 00000080:00010000:2.0:1767316275.912623:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x43c:0x0](ffff9d1f885d1988) for lock 0xa4df836b16491228 00000080:00010000:2.0:1767316275.912641:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00000080:00010000:2.0:1767316275.912644:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491228 from it ffffb2814198fb20 00010000:00010000:2.0:1767316275.912647:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.912655:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200008:2.0:1767316275.912786:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d135.replay-single:[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316275.912793:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988),name=d135.replay-single 00000002:00010000:2.0:1767316275.912812:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x43c:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.912819:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.912830:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.912836:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.912840:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00000080:00010000:2.0:1767316275.912843:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x43c:0x0](ffff9d1f885d1988) for lock 0xa4df836b16491228 00000080:00010000:2.0:1767316275.912847:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491228 from it ffffb2814198fcb8 00010000:00010000:2.0:1767316275.912850:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.912857:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.912905:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316275.912912:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.912924:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.912932:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.912938:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.912942:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316275.912945:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491221 00000080:00010000:2.0:1767316275.912949:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491221 from it ffffb2814198fbf8 00010000:00010000:2.0:1767316275.912952:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.912959:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.912968:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.912974:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=266 00000080:00200000:2.0:1767316275.912979:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 44 00000080:00200008:2.0:1767316275.912987:0:46893:0:(file.c:1001:ll_file_open()) START file d135.replay-single:[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags 704400 00000002:00010000:2.0:1767316275.913001:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x43c:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1767316275.913048:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b1649122f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.913056:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b1649122f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000020:00100000:2.0:1767316275.913067:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1767316275.913073:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4700 x1853163904558720/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/133032 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1767316275.913094:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b1649122f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.913421:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904558720:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.913606:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.918055:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904558720:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.918333:0:46893:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b1649122f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.918348:0:46893:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b1649122f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.918354:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b1649122f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.918361:0:46893:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b1649122f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.918368:0:46893:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b1649122f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.918375:0:46893:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f909f3e00 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:2.0:1767316275.918379:0:46893:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b1649122f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.918385:0:46893:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b1649122f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.918399:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b1649122f lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.918406:0:46893:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1767316275.918429:0:46893:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b1649122f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.918435:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b1649122f lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00000002:00100000:2.0:1767316275.918457:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9d1f890e4700 x1853163904558720/t51539607586(51539607586) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1767316275.918470:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4700 x1853163904558720/t51539607586(51539607586) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1767316275.918482:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316275.918504:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x43c:0x0], updating i_size 12800 00000002:00100000:2.0:1767316275.918520:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4700 x1853163904558720/t51539607586(51539607586) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1767316275.918596:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d135.replay-single:[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316275.918603:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988),name=d135.replay-single 00000002:00010000:2.0:1767316275.918626:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x43c:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.918640:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.918652:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.918660:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.918666:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00000080:00010000:2.0:1767316275.918670:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x43c:0x0](ffff9d1f885d1988) for lock 0xa4df836b16491228 00000080:00010000:2.0:1767316275.918675:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491228 from it ffffb2814198fcd8 00010000:00010000:2.0:1767316275.918777:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.918791:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.918822:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) pos/size0/12800 32bit_api 0 00000002:00010000:2.0:1767316275.918834:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x43c:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: readdir flags 0 00010000:00010000:2.0:1767316275.918841:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.918848:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1767316275.919665:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904558848:192.168.202.140@tcp:37:rm.0 00000100:00100000:2.0:1767316275.919804:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.930574:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904558848:192.168.202.140@tcp:37:rm.0 00010000:00010000:2.0:1767316275.931558:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.931576:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200008:2.0:1767316275.931650:0:46893:0:(file.c:424:ll_file_release()) START file d135.replay-single:[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags 704000 00000002:00100000:2.0:1767316275.931812:0:46893:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f890e4700 x1853163904558720/t51539607586(51539607586) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1767316275.931825:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1767316275.931831:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4e00 x1853163904559104/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/1800 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1767316275.931874:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904559104:192.168.202.140@tcp:35:rm.0 00000100:00100000:2.0:1767316275.932033:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.935550:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904559104:192.168.202.140@tcp:35:rm.0 00000002:00100000:2.0:1767316275.935585:0:46893:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f890e4700 x1853163904558720/t51539607586(51539607586) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1767316275.935663:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316275.935728:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.935747:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.935760:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.935768:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.935773:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316275.935776:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491221 00000080:00010000:2.0:1767316275.935783:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491221 from it ffffb2814198fbf8 00010000:00010000:2.0:1767316275.935787:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.935795:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.935810:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.935817:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d135.replay-single, flags=266 00000080:00200000:2.0:1767316275.935824:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 44 00000080:00200008:2.0:1767316275.935846:0:46893:0:(file.c:1001:ll_file_open()) START file d135.replay-single:[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags 704400 00000002:00010000:2.0:1767316275.935859:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x43c:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:2.0:1767316275.935906:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.935913:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000020:00100000:2.0:1767316275.935924:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1767316275.935930:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4e00 x1853163904559360/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/133032 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1767316275.935939:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.935957:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904559360:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.936763:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.942387:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4e00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904559360:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.942430:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b0f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.942443:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b0f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.942455:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9b0f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.942465:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b0f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.942481:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=200002b, status=0 req@ffff9d1f890e4e00 x1853163904559360/t51539607588(51539607588) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1767316275.942511:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4e00 x1853163904559360/t51539607588(51539607588) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.942524:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b0f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.942535:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 16 res 8589937617/1084 (0 0) 00000080:00200000:2.0:1767316275.942542:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316275.942562:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x43c:0x0], updating i_size 12800 00000080:00010000:2.0:1767316275.942567:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x43c:0x0](ffff9d1f885d1988) for lock 0xa4df836b16491236 00000080:00010000:2.0:1767316275.942575:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491236 from it ffffb2814198fbd0 00010000:00010000:2.0:1767316275.942579:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9b0f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.942589:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b0f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.942612:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f890e4e00 x1853163904559360/t51539607588(51539607588) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1767316275.942717:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d135.replay-single:[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316275.942726:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988),name=d135.replay-single 00000002:00010000:2.0:1767316275.942746:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x43c:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.942756:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.942766:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.942784:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316275.942790:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00000080:00010000:2.0:1767316275.942793:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x43c:0x0](ffff9d1f885d1988) for lock 0xa4df836b16491228 00000080:00010000:2.0:1767316275.942798:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491228 from it ffffb2814198fcd8 00010000:00010000:2.0:1767316275.942801:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.942808:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316275.942838:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) pos/size0/12800 32bit_api 0 00000080:00200000:2.0:1767316275.942897:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) pos/size18446744073709551614/12800 32bit_api 0 00000080:00200000:2.0:1767316275.942919:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.942932:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.13, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316275.942937:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.13, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316275.942949:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.13,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.942991:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.943427:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.943440:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.943465:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904559488:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.943608:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.947979:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904559488:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.948282:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.948303:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.948312:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.948323:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.948334:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.948346:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.948354:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.948360:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.948372:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e4700 x1853163904559488/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.948384:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316275.948419:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.948430:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1098 (0 0) 00000080:00200000:2.0:1767316275.948437:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316275.948482:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x44a:0x0](ffff9d1f885d5348): rc = 0 00000080:00200000:2.0:1767316275.948498:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x44a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316275.948504:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x44a:0x0](ffff9d1f885d5348) for lock 0xa4df836b1649123d 00000080:00010000:2.0:1767316275.948515:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x44a:0x0](ffff9d1f885d5348) 00000080:00010000:2.0:1767316275.948518:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649123d from it ffffb2814198fdf8 00010000:00010000:2.0:1767316275.948522:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.948529:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.948564:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316275.948573:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.13, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316275.948602:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.948620:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b0f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.948798:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.948810:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x44a:0x0]: losing layout lock 00010000:00010000:2.0:1767316275.948851:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.948858:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.948865:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1767316275.948868:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649123d lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x44a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9b32 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.948884:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491228 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9af3 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000020:00100000:2.0:1767316275.948901:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316275.948909:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4700 x1853163904559744/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316275.948934:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904559744:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316275.949042:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.953822:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904559744:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316275.953878:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x44a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316275.953891:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316275.953918:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x44a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316275.953925:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x44a:0x0](ffff9d1f885d5348) 00000080:00200000:2.0:1767316275.954012:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.954029:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.19, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316275.954034:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.19, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316275.954053:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.19,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.954099:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.955374:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.955388:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.955427:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904559872:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.955524:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.961079:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904559872:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.961102:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.961132:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.961140:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.961151:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.961161:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.961180:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.961189:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.961206:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.961218:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e4a80 x1853163904559872/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.961228:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316275.961240:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.961249:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1104 (0 0) 00000080:00200000:2.0:1767316275.961255:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316275.961292:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x450:0x0](ffff9d1f88476c08): rc = 0 00000080:00200000:2.0:1767316275.961304:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x450:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316275.961308:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x450:0x0](ffff9d1f88476c08) for lock 0xa4df836b16491244 00000080:00010000:2.0:1767316275.961317:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x450:0x0](ffff9d1f88476c08) 00000080:00010000:2.0:1767316275.961320:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491244 from it ffffb2814198fdf8 00010000:00010000:2.0:1767316275.961323:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.961331:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.961362:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316275.961370:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.19, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316275.961408:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.961432:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x450:0x0]: losing layout lock 00010000:00010000:2.0:1767316275.961467:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.961474:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316275.961476:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491244 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x450:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9b6a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316275.961501:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316275.961508:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4a80 x1853163904560128/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316275.961528:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904560128:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316275.961608:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.966626:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4a80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904560128:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316275.966709:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x450:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316275.966724:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316275.966750:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x450:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316275.966757:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x450:0x0](ffff9d1f88476c08) 00000080:00200000:2.0:1767316275.966836:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.966851:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.5, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316275.966856:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.5, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316275.966876:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.5,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.966924:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.966934:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.966957:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.966976:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7480 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904560256:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.967063:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.971531:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7480 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904560256:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.971554:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.971574:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.971582:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.971593:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.971602:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.971613:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.971620:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.971628:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.971651:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e7480 x1853163904560256/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.971662:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316275.971675:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.971719:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1090 (0 0) 00000080:00200000:2.0:1767316275.971725:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316275.971762:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x442:0x0](ffff9d1f88581988): rc = 0 00000080:00200000:2.0:1767316275.971773:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x442:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316275.971777:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x442:0x0](ffff9d1f88581988) for lock 0xa4df836b1649124b 00000080:00010000:2.0:1767316275.971786:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x442:0x0](ffff9d1f88581988) 00000080:00010000:2.0:1767316275.971789:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649124b from it ffffb2814198fdf8 00010000:00010000:2.0:1767316275.971792:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.971801:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.971835:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316275.971842:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.5, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316275.971868:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.971881:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x442:0x0]: losing layout lock 00010000:00010000:2.0:1767316275.971917:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.971923:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316275.971926:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649124b lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x442:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9ba2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316275.971949:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316275.971955:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e7480 x1853163904560512/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316275.971975:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7480 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904560512:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316275.972059:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.978566:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7480 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904560512:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316275.978624:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x442:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316275.978636:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316275.978661:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x442:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316275.978669:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x442:0x0](ffff9d1f88581988) 00000080:00200000:2.0:1767316275.978776:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.978792:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.2, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316275.978797:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.2, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316275.978818:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.2,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.978866:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.978875:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.978884:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.978920:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904560640:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.978976:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.983087:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904560640:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.983121:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.983143:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.983153:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.983165:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.983187:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.983200:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.983208:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.983215:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.983227:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e4380 x1853163904560640/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.983254:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316275.983267:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.983278:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1087 (0 0) 00000080:00200000:2.0:1767316275.983285:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316275.983329:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x43f:0x0](ffff9d1f885d4b08): rc = 0 00000080:00200000:2.0:1767316275.983343:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x43f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316275.983349:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x43f:0x0](ffff9d1f885d4b08) for lock 0xa4df836b16491252 00000080:00010000:2.0:1767316275.983359:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x43f:0x0](ffff9d1f885d4b08) 00000080:00010000:2.0:1767316275.983363:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491252 from it ffffb2814198fdf8 00010000:00010000:2.0:1767316275.983367:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.983375:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.983422:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316275.983431:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.2, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316275.983463:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.983476:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x43f:0x0]: losing layout lock 00010000:00010000:2.0:1767316275.983514:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.983521:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316275.983532:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491252 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x43f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9bda expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316275.983557:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316275.983564:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e4380 x1853163904560896/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316275.983586:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904560896:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316275.983716:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.989405:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e4380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904560896:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316275.989466:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x43f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316275.989479:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316275.989505:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x43f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316275.989513:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x43f:0x0](ffff9d1f885d4b08) 00000080:00200000:2.0:1767316275.989588:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.989604:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.8, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316275.989608:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.8, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316275.989629:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.8,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.989720:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.989729:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.989738:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316275.989758:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904561024:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316275.989830:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.995236:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904561024:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316275.995264:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.995285:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.995294:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.995305:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.995316:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.995327:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.995335:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316275.995342:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316275.995354:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e6a00 x1853163904561024/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316275.995365:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316275.995403:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.995414:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1093 (0 0) 00000080:00200000:2.0:1767316275.995421:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316275.995466:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x445:0x0](ffff9d1f88584b08): rc = 0 00000080:00200000:2.0:1767316275.995480:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x445:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316275.995485:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x445:0x0](ffff9d1f88584b08) for lock 0xa4df836b16491259 00000080:00010000:2.0:1767316275.995494:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x445:0x0](ffff9d1f88584b08) 00000080:00010000:2.0:1767316275.995497:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491259 from it ffffb2814198fdf8 00010000:00010000:2.0:1767316275.995501:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.995508:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.995538:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316275.995545:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.8, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316275.995573:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316275.995586:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x445:0x0]: losing layout lock 00010000:00010000:2.0:1767316275.995621:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316275.995627:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316275.995630:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491259 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x445:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9c12 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316275.995666:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316275.995672:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e6a00 x1853163904561280/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316275.995737:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904561280:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316275.995832:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316275.999657:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6a00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904561280:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316275.999753:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x445:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316275.999767:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316275.999791:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x445:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316275.999798:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x445:0x0](ffff9d1f88584b08) 00000080:00200000:2.0:1767316275.999871:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316275.999886:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.14, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316275.999891:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.14, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316275.999911:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.14,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316275.999963:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.999973:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316275.999982:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316276.000008:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904561408:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316276.000068:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.005654:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904561408:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316276.005759:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.005779:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.005786:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.005798:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.005809:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.005820:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.005827:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.005834:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.005844:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e7100 x1853163904561408/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.005855:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316276.005877:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.005886:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1099 (0 0) 00000080:00200000:2.0:1767316276.005892:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316276.005931:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x44b:0x0](ffff9d1f885c1988): rc = 0 00000080:00200000:2.0:1767316276.005944:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x44b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316276.005949:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x44b:0x0](ffff9d1f885c1988) for lock 0xa4df836b16491260 00000080:00010000:2.0:1767316276.005958:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x44b:0x0](ffff9d1f885c1988) 00000080:00010000:2.0:1767316276.005962:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491260 from it ffffb2814198fdf8 00010000:00010000:2.0:1767316276.005965:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.005974:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.006017:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316276.006025:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.14, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316276.006062:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.006086:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x44b:0x0]: losing layout lock 00010000:00010000:2.0:1767316276.006128:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.006135:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316276.006138:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491260 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x44b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9c4a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316276.006187:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316276.006194:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e7100 x1853163904561664/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316276.006217:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904561664:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316276.006289:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.012062:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904561664:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316276.012128:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x44b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316276.012141:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316276.012167:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x44b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.012186:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x44b:0x0](ffff9d1f885c1988) 00000080:00200000:2.0:1767316276.012264:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.012279:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.10, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316276.012283:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.10, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316276.012303:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.10,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.012350:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.012359:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.012368:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316276.012386:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5f80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904561792:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316276.012487:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.017766:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5f80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904561792:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316276.017808:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.017832:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.017842:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.017854:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.017864:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.017876:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.017884:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.017892:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.017905:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e5f80 x1853163904561792/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.017917:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316276.017942:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.017953:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1095 (0 0) 00000080:00200000:2.0:1767316276.017959:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316276.018019:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x447:0x0](ffff9d1f88583248): rc = 0 00000080:00200000:2.0:1767316276.018035:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x447:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316276.018041:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x447:0x0](ffff9d1f88583248) for lock 0xa4df836b16491267 00000080:00010000:2.0:1767316276.018053:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x447:0x0](ffff9d1f88583248) 00000080:00010000:2.0:1767316276.018056:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491267 from it ffffb2814198fdf8 00010000:00010000:2.0:1767316276.018060:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.018079:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.018120:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316276.018129:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.10, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316276.018160:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.018174:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x447:0x0]: losing layout lock 00010000:00010000:2.0:1767316276.018211:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.018218:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316276.018221:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491267 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x447:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9c82 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316276.018261:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316276.018267:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e5f80 x1853163904562048/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316276.018288:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5f80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904562048:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316276.018472:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f967593c0 going to sleep for 11 seconds 00000100:00100000:3.0:1767316276.024521:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5f80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904562048:192.168.202.140@tcp:36:rm.0 00000080:00200000:3.0:1767316276.024587:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x447:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1767316276.024601:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1767316276.024626:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x447:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1767316276.024633:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x447:0x0](ffff9d1f88583248) 00000080:00200000:3.0:1767316276.024748:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:3.0:1767316276.024766:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.6, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:3.0:1767316276.024771:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.6, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:3.0:1767316276.024796:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.6,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316276.024849:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316276.024859:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316276.024868:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:3.0:1767316276.024905:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904562176:192.168.202.140@tcp:101:rm.0 00000100:00100000:3.0:1767316276.024986:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f882b2c00 going to sleep for 11 seconds 00000100:00100000:3.0:1767316276.030140:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904562176:192.168.202.140@tcp:101:rm.0 00010000:00010000:3.0:1767316276.030170:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.030205:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.030213:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.030225:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.030238:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.030251:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.030258:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.030266:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1767316276.030279:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f92f73480 x1853163904562176/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1767316276.030304:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:3.0:1767316276.030318:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316276.030328:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1091 (0 0) 00000080:00200000:3.0:1767316276.030334:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1767316276.030382:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x443:0x0](ffff9d1f88580908): rc = 0 00000080:00200000:3.0:1767316276.030407:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x443:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1767316276.030414:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x443:0x0](ffff9d1f88580908) for lock 0xa4df836b1649126e 00000080:00010000:3.0:1767316276.030426:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x443:0x0](ffff9d1f88580908) 00000080:00010000:3.0:1767316276.030430:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649126e from it ffffb2814198fdf8 00010000:00010000:3.0:1767316276.030433:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316276.030441:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1767316276.030483:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:3.0:1767316276.030493:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.6, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:3.0:1767316276.030530:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1767316276.030545:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x443:0x0]: losing layout lock 00010000:00010000:3.0:1767316276.030584:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316276.030590:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1767316276.030600:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b1649126e lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x443:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9cba expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:3.0:1767316276.030630:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1767316276.030637:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f92f73480 x1853163904562432/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767316276.030659:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904562432:192.168.202.140@tcp:36:rm.0 00000100:00100000:3.0:1767316276.030784:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f882b2c00 going to sleep for 11 seconds 00000100:00100000:3.0:1767316276.035127:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904562432:192.168.202.140@tcp:36:rm.0 00000080:00200000:3.0:1767316276.035201:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x443:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1767316276.035215:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1767316276.035245:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x443:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1767316276.035252:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x443:0x0](ffff9d1f88580908) 00000080:00200000:3.0:1767316276.035338:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:3.0:1767316276.035355:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.1, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:3.0:1767316276.035362:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.1, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:3.0:1767316276.035387:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.1,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316276.035454:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316276.035465:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316276.035474:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:3.0:1767316276.035494:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904562560:192.168.202.140@tcp:101:rm.0 00000100:00100000:3.0:1767316276.035557:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f882b2c00 going to sleep for 11 seconds 00000100:00100000:3.0:1767316276.043636:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904562560:192.168.202.140@tcp:101:rm.0 00010000:00010000:3.0:1767316276.043663:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.043748:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.043757:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.043767:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.043778:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.043789:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.043797:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.043804:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1767316276.043816:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f92f71180 x1853163904562560/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1767316276.043827:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:3.0:1767316276.043855:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316276.043866:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1086 (0 0) 00000080:00200000:3.0:1767316276.043873:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1767316276.043913:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x43e:0x0](ffff9d1f885d63c8): rc = 0 00000080:00200000:3.0:1767316276.043926:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x43e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1767316276.043931:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x43e:0x0](ffff9d1f885d63c8) for lock 0xa4df836b16491275 00000080:00010000:3.0:1767316276.043940:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x43e:0x0](ffff9d1f885d63c8) 00000080:00010000:3.0:1767316276.043944:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491275 from it ffffb2814198fdf8 00010000:00010000:3.0:1767316276.043947:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316276.043955:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1767316276.043984:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:3.0:1767316276.043993:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.1, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:3.0:1767316276.044034:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1767316276.044048:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x43e:0x0]: losing layout lock 00010000:00010000:3.0:1767316276.044275:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316276.044282:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1767316276.044285:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b16491275 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x43e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9cf2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:3.0:1767316276.044321:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1767316276.044328:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f92f71180 x1853163904562816/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767316276.044350:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904562816:192.168.202.140@tcp:36:rm.0 00000100:00100000:3.0:1767316276.044497:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f882b2c00 going to sleep for 11 seconds 00000100:00100000:3.0:1767316276.049281:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904562816:192.168.202.140@tcp:36:rm.0 00000080:00200000:3.0:1767316276.049345:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x43e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1767316276.049359:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1767316276.049388:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x43e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1767316276.049407:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x43e:0x0](ffff9d1f885d63c8) 00000080:00200000:3.0:1767316276.049490:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:3.0:1767316276.049505:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.17, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:3.0:1767316276.049510:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.17, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:3.0:1767316276.049534:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.17,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316276.049586:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316276.049595:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316276.049605:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:3.0:1767316276.049623:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904562944:192.168.202.140@tcp:101:rm.0 00000100:00100000:3.0:1767316276.049861:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f882b2c00 going to sleep for 11 seconds 00000100:00100000:3.0:1767316276.055374:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904562944:192.168.202.140@tcp:101:rm.0 00010000:00010000:3.0:1767316276.055434:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.055455:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.055464:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.055476:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.055488:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.055500:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.055509:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.055516:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1767316276.055529:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f92f73800 x1853163904562944/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1767316276.055542:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:3.0:1767316276.055565:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316276.055575:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1102 (0 0) 00000080:00200000:3.0:1767316276.055581:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1767316276.055622:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x44e:0x0](ffff9d1f884721c8): rc = 0 00000080:00200000:3.0:1767316276.055634:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x44e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1767316276.055639:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x44e:0x0](ffff9d1f884721c8) for lock 0xa4df836b1649127c 00000080:00010000:3.0:1767316276.055650:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x44e:0x0](ffff9d1f884721c8) 00000080:00010000:3.0:1767316276.055653:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649127c from it ffffb2814198fdf8 00010000:00010000:3.0:1767316276.055657:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316276.055665:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1767316276.055732:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:3.0:1767316276.055741:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.17, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:3.0:1767316276.055772:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1767316276.055785:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x44e:0x0]: losing layout lock 00010000:00010000:3.0:1767316276.055823:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316276.055830:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1767316276.055833:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649127c lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x44e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9d2a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:3.0:1767316276.055872:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1767316276.055879:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f92f73800 x1853163904563200/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767316276.055905:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904563200:192.168.202.140@tcp:36:rm.0 00000100:00100000:3.0:1767316276.055981:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f882b2c00 going to sleep for 11 seconds 00000100:00100000:3.0:1767316276.060516:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904563200:192.168.202.140@tcp:36:rm.0 00000080:00200000:3.0:1767316276.060575:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x44e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1767316276.060588:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1767316276.060615:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x44e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1767316276.060622:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x44e:0x0](ffff9d1f884721c8) 00000080:00200000:3.0:1767316276.060739:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:3.0:1767316276.060755:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.15, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:3.0:1767316276.060759:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.15, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:3.0:1767316276.060777:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.15,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316276.060825:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316276.060834:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316276.060843:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:3.0:1767316276.060861:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904563328:192.168.202.140@tcp:101:rm.0 00000100:00100000:3.0:1767316276.060924:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f882b2c00 going to sleep for 11 seconds 00000100:00100000:3.0:1767316276.065324:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904563328:192.168.202.140@tcp:101:rm.0 00010000:00010000:3.0:1767316276.065361:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.065382:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.065400:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.065412:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.065423:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.065434:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.065441:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.065448:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1767316276.065459:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f92f70700 x1853163904563328/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1767316276.065471:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:3.0:1767316276.065495:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316276.065507:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1100 (0 0) 00000080:00200000:3.0:1767316276.065513:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1767316276.065557:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x44c:0x0](ffff9d1f885c21c8): rc = 0 00000080:00200000:3.0:1767316276.065569:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x44c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1767316276.065574:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x44c:0x0](ffff9d1f885c21c8) for lock 0xa4df836b16491283 00000080:00010000:3.0:1767316276.065583:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x44c:0x0](ffff9d1f885c21c8) 00000080:00010000:3.0:1767316276.065586:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491283 from it ffffb2814198fdf8 00010000:00010000:3.0:1767316276.065589:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316276.065597:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1767316276.065627:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:3.0:1767316276.065635:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.15, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:3.0:1767316276.065668:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:3.0:1767316276.065749:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x44c:0x0]: losing layout lock 00010000:00010000:3.0:1767316276.065787:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1767316276.065794:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1767316276.065797:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b384c00/0xa4df836b16491283 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x44c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9d62 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:3.0:1767316276.065834:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1767316276.065842:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f92f70700 x1853163904563584/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767316276.065867:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904563584:192.168.202.140@tcp:36:rm.0 00000100:00100000:3.0:1767316276.065959:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f882b2c00 going to sleep for 11 seconds 00000100:00100000:3.0:1767316276.070055:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904563584:192.168.202.140@tcp:36:rm.0 00000080:00200000:3.0:1767316276.070117:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x44c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1767316276.070130:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1767316276.070156:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x44c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1767316276.070164:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x44c:0x0](ffff9d1f885c21c8) 00000080:00200000:3.0:1767316276.070263:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:3.0:1767316276.070281:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.9, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:3.0:1767316276.070287:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.9, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:3.0:1767316276.070308:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.9,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767316276.070357:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316276.070367:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:3.0:1767316276.070377:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:3.0:1767316276.070406:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71f80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904563712:192.168.202.140@tcp:101:rm.0 00000100:00100000:3.0:1767316276.070484:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f882b2c00 going to sleep for 11 seconds 00000100:00100000:3.0:1767316276.074906:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71f80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904563712:192.168.202.140@tcp:101:rm.0 00010000:00010000:3.0:1767316276.074950:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.075362:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.075372:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.075383:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.075405:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.075418:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.075424:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.075430:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.075442:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f92f71f80 x1853163904563712/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.075461:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316276.075476:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.075487:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1094 (0 0) 00000080:00200000:2.0:1767316276.075494:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316276.075546:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x446:0x0](ffff9d1f885800c8): rc = 0 00000080:00200000:2.0:1767316276.075560:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x446:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316276.075565:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x446:0x0](ffff9d1f885800c8) for lock 0xa4df836b1649128a 00000080:00010000:2.0:1767316276.075575:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x446:0x0](ffff9d1f885800c8) 00000080:00010000:2.0:1767316276.075578:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649128a from it ffffb2814198fdf8 00010000:00010000:2.0:1767316276.075581:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.075588:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.075621:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316276.075630:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.9, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316276.075662:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.075703:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x446:0x0]: losing layout lock 00010000:00010000:2.0:1767316276.075747:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.075753:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316276.075762:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385e00/0xa4df836b1649128a lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x446:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9d9a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316276.075790:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316276.075797:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e7800 x1853163904563968/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316276.075818:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7800 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904563968:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316276.075961:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.082607:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7800 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904563968:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316276.082674:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x446:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316276.082772:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316276.082806:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x446:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.082815:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x446:0x0](ffff9d1f885800c8) 00000080:00200000:2.0:1767316276.082915:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.082932:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.3, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316276.082937:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.3, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316276.082957:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.3,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.083152:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.083161:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.083170:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316276.083372:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7b80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904564096:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316276.083529:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.089486:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7b80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904564096:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316276.089512:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.089534:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.089541:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.089552:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.089563:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.089573:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.089581:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.089588:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.089600:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e7b80 x1853163904564096/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.089612:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316276.089643:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.089653:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1088 (0 0) 00000080:00200000:2.0:1767316276.089659:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316276.089774:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x440:0x0](ffff9d1f885d1148): rc = 0 00000080:00200000:2.0:1767316276.089787:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x440:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316276.089793:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x440:0x0](ffff9d1f885d1148) for lock 0xa4df836b16491291 00000080:00010000:2.0:1767316276.089802:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x440:0x0](ffff9d1f885d1148) 00000080:00010000:2.0:1767316276.089806:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491291 from it ffffb2814198fdf8 00010000:00010000:2.0:1767316276.089809:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.089817:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.089851:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316276.089859:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.3, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316276.089892:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.089906:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x440:0x0]: losing layout lock 00010000:00010000:2.0:1767316276.089942:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.089950:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316276.089952:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b16491291 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x440:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9dd2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316276.089989:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316276.089995:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e7b80 x1853163904564352/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316276.090285:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e7b80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904564352:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316276.090443:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.099225:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e7b80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904564352:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316276.099282:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x440:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316276.099295:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316276.099322:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x440:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.099329:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x440:0x0](ffff9d1f885d1148) 00000080:00200000:2.0:1767316276.099426:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.099444:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.4, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316276.099450:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.4, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316276.099472:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.4,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.099534:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.099544:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.099553:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316276.099571:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5880 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904564480:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316276.099645:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.106129:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5880 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904564480:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316276.106169:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.106202:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.106209:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.106221:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.106233:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.106246:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.106254:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.106262:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.106275:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e5880 x1853163904564480/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.106287:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316276.106310:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.106320:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1089 (0 0) 00000080:00200000:2.0:1767316276.106326:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316276.106370:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x441:0x0](ffff9d1f88585348): rc = 0 00000080:00200000:2.0:1767316276.106384:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x441:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316276.106397:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x441:0x0](ffff9d1f88585348) for lock 0xa4df836b16491298 00000080:00010000:2.0:1767316276.106407:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x441:0x0](ffff9d1f88585348) 00000080:00010000:2.0:1767316276.106410:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491298 from it ffffb2814198fdf8 00010000:00010000:2.0:1767316276.106413:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.106421:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.106454:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316276.106461:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.4, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316276.106492:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.106505:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x441:0x0]: losing layout lock 00010000:00010000:2.0:1767316276.106538:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.106544:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316276.106547:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491298 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x441:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9e0a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316276.106582:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316276.106589:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e5880 x1853163904564736/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316276.106612:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5880 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904564736:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316276.106725:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.113317:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5880 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904564736:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316276.113381:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x441:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316276.113413:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316276.113442:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x441:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.113449:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x441:0x0](ffff9d1f88585348) 00000080:00200000:2.0:1767316276.113536:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.113554:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.20, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316276.113559:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.20, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316276.113582:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.20,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.113633:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.113643:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.113652:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316276.113672:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5500 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904564864:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316276.113960:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.118565:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5500 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904564864:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316276.118617:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.118640:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.118648:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.118659:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.118670:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.118718:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.118727:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.118734:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.118746:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e5500 x1853163904564864/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.118759:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316276.118781:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.118791:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1105 (0 0) 00000080:00200000:2.0:1767316276.118798:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316276.118843:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x451:0x0](ffff9d1f884763c8): rc = 0 00000080:00200000:2.0:1767316276.118856:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x451:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316276.118861:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x451:0x0](ffff9d1f884763c8) for lock 0xa4df836b1649129f 00000080:00010000:2.0:1767316276.118871:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x451:0x0](ffff9d1f884763c8) 00000080:00010000:2.0:1767316276.118875:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649129f from it ffffb2814198fdf8 00010000:00010000:2.0:1767316276.118878:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.118887:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.118918:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316276.118925:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.20, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316276.118955:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.118970:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x451:0x0]: losing layout lock 00010000:00010000:2.0:1767316276.119002:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.119010:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316276.119012:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b1649129f lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x451:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9e42 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316276.119048:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316276.119055:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e5500 x1853163904565120/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316276.119079:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e5500 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904565120:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316276.119163:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.127871:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e5500 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904565120:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316276.127945:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x451:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316276.127961:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316276.127999:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x451:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.128008:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x451:0x0](ffff9d1f884763c8) 00000080:00200000:2.0:1767316276.128118:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.128135:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.18, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316276.128140:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.18, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316276.128162:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.18,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.128235:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.128246:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.128256:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316276.128278:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904565248:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316276.129627:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.134045:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904565248:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316276.134097:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.134121:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.134129:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.134140:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.134151:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.134162:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.134170:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.134177:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.134190:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f890e6d80 x1853163904565248/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.134201:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316276.134224:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.134235:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1103 (0 0) 00000080:00200000:2.0:1767316276.134242:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316276.134288:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x44f:0x0](ffff9d1f88472a08): rc = 0 00000080:00200000:2.0:1767316276.134300:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x44f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316276.134305:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x44f:0x0](ffff9d1f88472a08) for lock 0xa4df836b164912a6 00000080:00010000:2.0:1767316276.134314:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x44f:0x0](ffff9d1f88472a08) 00000080:00010000:2.0:1767316276.134317:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912a6 from it ffffb2814198fdf8 00010000:00010000:2.0:1767316276.134320:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.134338:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.134371:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316276.134379:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.18, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316276.134416:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.134430:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x44f:0x0]: losing layout lock 00010000:00010000:2.0:1767316276.134465:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.134471:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316276.134474:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912a6 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x44f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9e7a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316276.134508:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316276.134515:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f890e6d80 x1853163904565504/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316276.134538:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904565504:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316276.134622:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.141089:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f890e6d80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904565504:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316276.141149:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x44f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316276.141163:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316276.141202:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x44f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.141211:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x44f:0x0](ffff9d1f88472a08) 00000080:00200000:2.0:1767316276.141288:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.141304:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.12, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316276.141308:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.12, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316276.141327:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.12,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.141370:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.141378:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.141385:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316276.141414:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904565632:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316276.141740:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.147948:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904565632:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316276.147990:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.148025:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.148034:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.148046:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.148057:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.148078:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.148086:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.148093:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.148104:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f967cc380 x1853163904565632/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.148116:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316276.148139:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.148149:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1097 (0 0) 00000080:00200000:2.0:1767316276.148157:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316276.148210:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x449:0x0](ffff9d1f88586c08): rc = 0 00000080:00200000:2.0:1767316276.148223:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x449:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316276.148229:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x449:0x0](ffff9d1f88586c08) for lock 0xa4df836b164912ad 00000080:00010000:2.0:1767316276.148238:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x449:0x0](ffff9d1f88586c08) 00000080:00010000:2.0:1767316276.148241:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912ad from it ffffb2814198fdf8 00010000:00010000:2.0:1767316276.148245:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.148253:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.148284:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316276.148291:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.12, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316276.148319:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.148333:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x449:0x0]: losing layout lock 00010000:00010000:2.0:1767316276.148366:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.148372:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316276.148375:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912ad lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x449:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9eb2 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316276.148423:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316276.148431:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f967cc380 x1853163904565888/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316276.148456:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904565888:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316276.148543:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00010000:00010000:0.0:1767316276.151047:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316276.154935:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904565888:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316276.155014:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x449:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316276.155027:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316276.155059:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x449:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.155073:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x449:0x0](ffff9d1f88586c08) 00000080:00200000:2.0:1767316276.155162:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.155189:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.16, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316276.155194:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.16, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316276.155218:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.16,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.155275:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.155284:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.155294:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316276.155314:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904566016:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316276.155397:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.160991:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904566016:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316276.161158:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.161341:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.161351:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.161362:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.161373:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.161386:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.161406:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.161413:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.161426:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f967cc000 x1853163904566016/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.161438:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316276.161461:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.161473:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1101 (0 0) 00000080:00200000:2.0:1767316276.161480:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316276.161528:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x44d:0x0](ffff9d1f885c0908): rc = 0 00000080:00200000:2.0:1767316276.161542:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x44d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316276.161548:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x44d:0x0](ffff9d1f885c0908) for lock 0xa4df836b164912b4 00000080:00010000:2.0:1767316276.161560:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x44d:0x0](ffff9d1f885c0908) 00000080:00010000:2.0:1767316276.161564:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912b4 from it ffffb2814198fdf8 00010000:00010000:2.0:1767316276.161568:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.161576:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.161608:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316276.161616:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.16, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316276.161647:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.161662:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x44d:0x0]: losing layout lock 00010000:00010000:2.0:1767316276.161731:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.161738:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316276.161741:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3a00/0xa4df836b164912b4 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x44d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9eea expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316276.161778:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316276.161786:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f967cc000 x1853163904566272/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316276.161811:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904566272:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316276.161954:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.168487:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc000 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904566272:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316276.168545:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x44d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316276.168558:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316276.168584:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x44d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.168591:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x44d:0x0](ffff9d1f885c0908) 00000080:00200000:2.0:1767316276.168670:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.168715:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.7, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316276.168720:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.7, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316276.168742:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.7,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.168794:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.168803:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.168812:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316276.168832:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cce00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904566400:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316276.168901:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759f00 going to sleep for 11 seconds 00000100:00100000:3.0:1767316276.172560:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cce00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904566400:192.168.202.140@tcp:101:rm.0 00010000:00010000:3.0:1767316276.172591:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.172616:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.172625:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.172638:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.172650:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.172661:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.172669:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767316276.172759:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1767316276.172773:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f967cce00 x1853163904566400/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1767316276.172799:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316276.172879:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.172898:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1092 (0 0) 00000080:00200000:2.0:1767316276.172905:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316276.172950:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x444:0x0](ffff9d1f88581148): rc = 0 00000080:00200000:2.0:1767316276.172964:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x444:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316276.172970:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x444:0x0](ffff9d1f88581148) for lock 0xa4df836b164912bb 00000080:00010000:2.0:1767316276.172980:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x444:0x0](ffff9d1f88581148) 00000080:00010000:2.0:1767316276.172984:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912bb from it ffffb2814198fdf8 00010000:00010000:2.0:1767316276.172988:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.173060:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.173296:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316276.173305:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.7, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316276.173338:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.173354:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x444:0x0]: losing layout lock 00010000:00010000:2.0:1767316276.173401:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.173409:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316276.173411:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b164912bb lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x444:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9f22 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316276.173439:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316276.173446:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f967cce00 x1853163904566656/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316276.173477:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cce00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904566656:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316276.173628:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759300 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.179504:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cce00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904566656:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316276.179561:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x444:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316276.179574:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1767316276.179602:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x444:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.179609:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x444:0x0](ffff9d1f88581148) 00000080:00200000:2.0:1767316276.179741:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.179759:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=file.11, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags=0 00000080:00200000:2.0:1767316276.179765:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=file.11, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), intent=getattr 00000002:00010000:2.0:1767316276.179788:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file.11,[0x0:0x0:0x0]) in obj [0x200000bd1:0x43c:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.179839:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.179848:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.179858:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316276.179880:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904566784:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316276.179944:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759300 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.184408:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904566784:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316276.184434:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.184456:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.184479:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.184491:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.184502:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.184515:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.184522:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.184530:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.184543:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f967cc700 x1853163904566784/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/640 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.184555:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316276.184569:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.184580:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1096 (0 0) 00000080:00200000:2.0:1767316276.184586:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x43c:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316276.184641:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x448:0x0](ffff9d1f88587448): rc = 0 00000080:00200000:2.0:1767316276.184656:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x448:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1767316276.184661:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x448:0x0](ffff9d1f88587448) for lock 0xa4df836b164912c2 00000080:00010000:2.0:1767316276.184670:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x448:0x0](ffff9d1f88587448) 00000080:00010000:2.0:1767316276.184673:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912c2 from it ffffb2814198fdf8 00010000:00010000:2.0:1767316276.184704:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.184713:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.184747:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), inode mode 41ed mask 3 00000080:00200000:2.0:1767316276.184756:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=file.11, dir=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00010000:00010000:2.0:1767316276.184784:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.184799:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x448:0x0]: losing layout lock 00010000:00010000:2.0:1767316276.184835:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.184843:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316276.184846:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b164912c2 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x448:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9f5a expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00100000:2.0:1767316276.184872:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316276.184878:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f967cc700 x1853163904567040/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 488/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316276.184899:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904567040:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316276.185041:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759300 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.195396:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc700 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904567040:192.168.202.140@tcp:36:rm.0 00000080:00200000:0.0:1767316276.195575:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x448:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1767316276.195589:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1767316276.195618:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x448:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1767316276.195625:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x448:0x0](ffff9d1f88587448) 00000080:00200008:0.0:1767316276.195745:0:46893:0:(file.c:424:ll_file_release()) START file d135.replay-single:[0x200000bd1:0x43c:0x0](ffff9d1f885d1988), flags 704000 00010000:00010000:0.0:1767316276.195771:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9b0f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.195819:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316276.195837:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.195846:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.195855:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.195863:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.195868:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316276.195872:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491221 00000080:00010000:0.0:1767316276.195879:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491221 from it ffffb2814198fc10 00010000:00010000:0.0:1767316276.195884:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.195891:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.195920:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316276.195935:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=0 00000080:00200000:0.0:1767316276.195939:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:0.0:1767316276.195951:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d135.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.196009:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.196017:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.196027:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316276.196053:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904567168:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316276.196874:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.201433:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904567168:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316276.201466:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.201842:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.201851:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.201863:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.201896:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.201909:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.201916:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.201923:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316276.201935:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f91032d80 x1853163904567168/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/584 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316276.201947:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.201956:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1084 (0 0) 00000080:00200000:0.0:1767316276.201963:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1767316276.201985:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x43c:0x0], updating i_size 12800 00000080:00200000:0.0:1767316276.201988:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x43c:0x0](ffff9d1f885d1988): rc = 0 00000080:00010000:0.0:1767316276.201994:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x43c:0x0](ffff9d1f885d1988) for lock 0xa4df836b164912c9 00000080:00010000:0.0:1767316276.202155:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00000080:00010000:0.0:1767316276.202158:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912c9 from it ffffb2814198fe00 00010000:00010000:0.0:1767316276.202162:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.202251:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.202289:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316276.202308:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.202318:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.202330:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.202336:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.202341:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316276.202344:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491221 00000080:00010000:0.0:1767316276.202348:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491221 from it ffffb2814198fd88 00010000:00010000:0.0:1767316276.202351:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.202358:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.202365:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 3 00000080:00200000:0.0:1767316276.202373:0:46893:0:(namei.c:2264:ll_rmdir()) VFS Op:name=d135.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348) 00010000:00010000:0.0:1767316276.202403:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.202428:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.202454:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9b0f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316276.202497:0:46893:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f890e4e00 x1853163904559360/t51539607588(51539607588) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1767316276.202509:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1767316276.202514:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91032d80 x1853163904567424/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/1800 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1767316276.202538:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904567424:192.168.202.140@tcp:35:rm.0 00000100:00100000:0.0:1767316276.202836:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.205873:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904567424:192.168.202.140@tcp:35:rm.0 00000002:00100000:0.0:1767316276.205906:0:46893:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f890e4e00 x1853163904559360/t51539607588(51539607588) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316291 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1767316276.205987:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.206008:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9b0f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.206016:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.206021:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1767316276.206024:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912c9 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9f92 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.206051:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3e00/0xa4df836b16491236 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x43c:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9b0f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.206063:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2c00/0xa4df836b16491221 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9ad7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000020:00100000:0.0:1767316276.206090:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1767316276.206097:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91032d80 x1853163904567680/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 512/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767316276.206119:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904567680:192.168.202.140@tcp:36:rm.0 00000100:00100000:0.0:1767316276.206189:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.211723:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904567680:192.168.202.140@tcp:36:rm.0 00000080:00200000:0.0:1767316276.211771:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x43c:0x0](ffff9d1f885d1988) 00000080:00200000:0.0:1767316276.211844:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316276.211865:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.211926:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.211935:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.211945:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316276.211965:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904567808:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316276.212038:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.216604:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904567808:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316276.216650:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.216663:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.216674:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.216731:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.216742:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.216750:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.216757:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316276.216767:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f91031880 x1853163904567808/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316276.216780:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.216794:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1767316276.216801:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1767316276.216830:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1767316276.216841:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164912d0 00000080:00010000:0.0:1767316276.216846:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316276.216850:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912d0 from it ffffb2814198fb58 00010000:00010000:0.0:1767316276.216853:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.216862:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.216886:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316276.216901:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d202.replay-single, flags=8 00000080:00200008:0.0:1767316276.216916:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1767316276.216921:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8),name=d202.replay-single 00000002:00010000:0.0:1767316276.216942:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x467:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.216951:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.216960:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.216966:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.216971:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) 00000080:00010000:0.0:1767316276.216973:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) for lock 0xa4df836b164910d8 00000080:00010000:0.0:1767316276.216976:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164910d8 from it ffffb2814198fcb8 00010000:00010000:0.0:1767316276.216979:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.216985:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.217221:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316276.217228:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.217234:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.217242:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.217248:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.217253:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316276.217255:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164912d0 00000080:00010000:0.0:1767316276.217259:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912d0 from it ffffb2814198fbf8 00010000:00010000:0.0:1767316276.217261:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.217269:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.217276:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316276.217280:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d202.replay-single, flags=266 00000080:00200000:0.0:1767316276.217285:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), inode mode 41ed mask 44 00000080:00200008:0.0:1767316276.217293:0:46893:0:(file.c:1001:ll_file_open()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), flags 704400 00000002:00010000:0.0:1767316276.217308:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x467:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:0.0:1767316276.217362:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912d7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.217369:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912d7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000020:00100000:0.0:1767316276.217400:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1767316276.217407:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91031880 x1853163904568064/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/133032 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1767316276.217417:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912d7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316276.217440:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904568064:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316276.217605:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.222254:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904568064:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316276.222284:0:46893:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912d7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.222298:0:46893:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912d7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.222304:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912d7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.222311:0:46893:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912d7 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.222319:0:46893:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912d7 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.222325:0:46893:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f93c85c00 already unused, calling callback (ffffffffc1905b40) 00010000:00010000:0.0:1767316276.222330:0:46893:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912d7 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.222337:0:46893:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912d7 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.222365:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912d7 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.222371:0:46893:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1767316276.222378:0:46893:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912d7 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.222384:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912d7 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 46893 initiator: MDT0 00000002:00100000:0.0:1767316276.222416:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9d1f91031880 x1853163904568064/t51539607611(51539607611) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316292 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1767316276.222429:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f91031880 x1853163904568064/t51539607611(51539607611) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316292 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1767316276.222443:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1767316276.222463:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x467:0x0], updating i_size 11776 00000002:00100000:0.0:1767316276.222480:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f91031880 x1853163904568064/t51539607611(51539607611) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316292 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1767316276.222564:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1767316276.222573:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8),name=d202.replay-single 00000002:00010000:0.0:1767316276.222594:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x467:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.222611:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.222623:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.222630:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.222642:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) 00000080:00010000:0.0:1767316276.222645:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) for lock 0xa4df836b164910d8 00000080:00010000:0.0:1767316276.222651:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164910d8 from it ffffb2814198fcd8 00010000:00010000:0.0:1767316276.222655:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.222662:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.222813:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8) pos/size0/11776 32bit_api 0 00000002:00010000:0.0:1767316276.222826:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x467:0x0]) in obj [0x200000bd1:0x467:0x0], intent: readdir flags 0 00010000:00010000:0.0:1767316276.222833:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.222841:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1767316276.223841:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904568192:192.168.202.140@tcp:37:rm.0 00000100:00100000:0.0:1767316276.223942:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.234828:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904568192:192.168.202.140@tcp:37:rm.0 00010000:00010000:0.0:1767316276.235512:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.235528:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000080:00200008:0.0:1767316276.235586:0:46893:0:(file.c:424:ll_file_release()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), flags 704000 00000002:00100000:0.0:1767316276.235649:0:46893:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f91031880 x1853163904568064/t51539607611(51539607611) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316292 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1767316276.235671:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1767316276.235707:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91033100 x1853163904568448/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/1800 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1767316276.235730:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904568448:192.168.202.140@tcp:35:rm.0 00000100:00100000:0.0:1767316276.235818:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.238466:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904568448:192.168.202.140@tcp:35:rm.0 00000002:00100000:0.0:1767316276.238506:0:46893:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f91031880 x1853163904568064/t51539607611(51539607611) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1767316276.238586:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316276.238604:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.238621:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.238634:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.238641:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.238647:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316276.238651:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164912d0 00000080:00010000:0.0:1767316276.238657:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912d0 from it ffffb2814198fbf8 00010000:00010000:0.0:1767316276.238661:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.238670:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.238759:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316276.238768:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d202.replay-single, flags=266 00000080:00200000:0.0:1767316276.238777:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), inode mode 41ed mask 44 00000080:00200008:0.0:1767316276.238786:0:46893:0:(file.c:1001:ll_file_open()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), flags 704400 00000002:00010000:0.0:1767316276.238802:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x467:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1767316276.238854:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.238862:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000020:00100000:0.0:1767316276.238873:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1767316276.238880:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91033100 x1853163904568704/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/133032 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1767316276.238889:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316276.238909:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904568704:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316276.239103:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.243053:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904568704:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316276.243099:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fd1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.243113:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fd1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.243127:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9fd1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.243153:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fd1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316276.243169:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=200002b, status=0 req@ffff9d1f91033100 x1853163904568704/t51539607613(51539607613) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316292 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1767316276.243349:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f91033100 x1853163904568704/t51539607613(51539607613) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316292 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316276.243363:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fd1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.243375:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 16 res 8589937617/1127 (0 0) 00000080:00200000:0.0:1767316276.243382:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1767316276.243414:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x467:0x0], updating i_size 11776 00000080:00010000:0.0:1767316276.243420:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) for lock 0xa4df836b164912de 00000080:00010000:0.0:1767316276.243428:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912de from it ffffb2814198fbd0 00010000:00010000:0.0:1767316276.243432:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9fd1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.243443:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fd1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316276.243469:0:46893:0:(mdc_request.c:810:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d1f91033100 x1853163904568704/t51539607613(51539607613) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316292 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1767316276.243550:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1767316276.243558:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8),name=d202.replay-single 00000002:00010000:0.0:1767316276.243579:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x467:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.243596:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.243607:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.243614:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.243621:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) 00000080:00010000:0.0:1767316276.243624:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) for lock 0xa4df836b164910d8 00000080:00010000:0.0:1767316276.243629:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164910d8 from it ffffb2814198fcd8 00010000:00010000:0.0:1767316276.243632:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.243640:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.243673:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8) pos/size0/11776 32bit_api 0 00000080:00200000:0.0:1767316276.243728:0:46893:0:(dir.c:354:ll_iterate()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8) pos/size18446744073709551614/11776 32bit_api 0 00000080:00200000:0.0:1767316276.243752:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), inode mode 41ed mask 1 00000080:00200000:0.0:1767316276.243767:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=f202.replay-single, dir=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), flags=0 00000080:00200000:0.0:1767316276.243771:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=f202.replay-single, dir=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), intent=getattr 00000002:00010000:0.0:1767316276.243782:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f202.replay-single,[0x0:0x0:0x0]) in obj [0x200000bd1:0x467:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.243823:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.243830:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.243838:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316276.243865:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904568832:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316276.244009:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.248448:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904568832:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316276.248473:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.248494:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.248502:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.248513:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.248524:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.248536:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.248543:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.248551:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316276.248562:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f91031880 x1853163904568832/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/824 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316276.248586:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1767316276.248598:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.248609:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1129 (0 0) 00000080:00200000:0.0:1767316276.248616:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x467:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1767316276.248658:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x469:0x0](ffff9d1f88473248): rc = 0 00000080:00200000:0.0:1767316276.248672:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x469:0x0]: layout version change: 4294967294 -> 2 00000080:00010000:0.0:1767316276.248740:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x469:0x0](ffff9d1f88473248) for lock 0xa4df836b164912e5 00000080:00010000:0.0:1767316276.248751:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x469:0x0](ffff9d1f88473248) 00000080:00010000:0.0:1767316276.248755:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912e5 from it ffffb2814198fdf8 00010000:00010000:0.0:1767316276.248758:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.248767:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316276.248794:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8), inode mode 41ed mask 3 00000080:00200000:0.0:1767316276.248802:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=f202.replay-single, dir=[0x200000bd1:0x467:0x0](ffff9d1f884742c8) 00010000:00010000:0.0:1767316276.248829:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.248845:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fd1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.248913:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316276.248930:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x469:0x0]: losing layout lock 00010000:00010000:0.0:1767316276.248960:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.248967:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.248973:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1767316276.248976:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912e5 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x469:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c9ff4 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.248990:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f3000/0xa4df836b164910d8 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c970a expref: -99 pid: 45532 timeout: 0 lvb_type: 0 00000020:00100000:0.0:1767316276.249147:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1767316276.249155:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91031880 x1853163904569088/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 504/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767316276.249241:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904569088:192.168.202.140@tcp:36:rm.0 00000100:00100000:0.0:1767316276.249371:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.258855:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904569088:192.168.202.140@tcp:36:rm.0 00000080:00200000:0.0:1767316276.258914:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x469:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1767316276.258928:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:0.0:1767316276.258937:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:0.0:1767316276.258956:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x469:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1767316276.258962:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x469:0x0](ffff9d1f88473248) 00000080:00200008:0.0:1767316276.259042:0:46893:0:(file.c:424:ll_file_release()) START file d202.replay-single:[0x200000bd1:0x467:0x0](ffff9d1f884742c8), flags 704000 00010000:00010000:0.0:1767316276.259075:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fd1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.259133:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316276.259149:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.259158:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.259168:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.259187:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.259193:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316276.259197:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164912d0 00000080:00010000:0.0:1767316276.259203:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912d0 from it ffffb2814198fc10 00010000:00010000:0.0:1767316276.259207:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.259215:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.259230:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316276.259244:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=d202.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=0 00000080:00200000:0.0:1767316276.259249:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=d202.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:0.0:1767316276.259260:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d202.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.259298:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.259305:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.259324:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316276.259343:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904569216:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316276.259409:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.263473:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904569216:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316276.263506:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.263526:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.263534:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.263547:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.263560:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.263573:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.263582:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.263589:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316276.263620:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f91030380 x1853163904569216/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/584 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316276.263633:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.263644:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1127 (0 0) 00000080:00200000:0.0:1767316276.263651:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1767316276.263702:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000bd1:0x467:0x0], updating i_size 11776 00000080:00200000:0.0:1767316276.263705:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x467:0x0](ffff9d1f884742c8): rc = 0 00000080:00010000:0.0:1767316276.263710:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) for lock 0xa4df836b164912ec 00000080:00010000:0.0:1767316276.263722:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x467:0x0](ffff9d1f884742c8) 00000080:00010000:0.0:1767316276.263725:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912ec from it ffffb2814198fe00 00010000:00010000:0.0:1767316276.263728:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.263736:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.263763:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316276.263779:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.263787:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.263797:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.263804:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.263815:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316276.263818:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164912d0 00000080:00010000:0.0:1767316276.263822:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912d0 from it ffffb2814198fd88 00010000:00010000:0.0:1767316276.263825:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.263832:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.263840:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 3 00000080:00200000:0.0:1767316276.263849:0:46893:0:(namei.c:2264:ll_rmdir()) VFS Op:name=d202.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348) 00010000:00010000:0.0:1767316276.263874:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.263897:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.263909:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8c9fd1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316276.263958:0:46893:0:(mdc_request.c:963:mdc_close()) @@@ matched open req@ffff9d1f91033100 x1853163904568704/t51539607613(51539607613) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316292 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1767316276.263969:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1767316276.263975:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91030380 x1853163904569472/t0(0) o35->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:23/10 lens 392/1800 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1767316276.264000:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904569472:192.168.202.140@tcp:35:rm.0 00000100:00100000:0.0:1767316276.264496:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.266927:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904569472:192.168.202.140@tcp:35:rm.0 00000002:00100000:0.0:1767316276.266969:0:46893:0:(mdc_request.c:831:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d1f91033100 x1853163904568704/t51539607613(51539607613) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 1824/608 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1767316276.267169:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.267318:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9fd1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.267325:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.267332:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1767316276.267335:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84600/0xa4df836b164912ec lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x467:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8ca02c expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.267348:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85c00/0xa4df836b164912de lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x467:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9fd1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.267364:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912d0 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8c9fb5 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000020:00100000:0.0:1767316276.267383:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1767316276.267398:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91030380 x1853163904569728/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 512/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767316276.267419:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904569728:192.168.202.140@tcp:36:rm.0 00000100:00100000:0.0:1767316276.267531:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.272886:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904569728:192.168.202.140@tcp:36:rm.0 00000080:00200000:0.0:1767316276.272959:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x467:0x0](ffff9d1f884742c8) 00000080:00200000:0.0:1767316276.273230:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316276.273251:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.273311:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.273321:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.273330:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316276.273350:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904569856:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316276.273443:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.276440:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904569856:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316276.276470:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.276483:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.276496:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.276507:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.276518:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.276547:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.276554:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316276.276567:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f91033100 x1853163904569856/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316276.276583:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.276598:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1767316276.276607:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1767316276.276636:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:0.0:1767316276.276643:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164912f3 00000080:00010000:0.0:1767316276.276649:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316276.276653:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912f3 from it ffffb2814198fb58 00010000:00010000:0.0:1767316276.276657:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.276667:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.276768:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316276.276795:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=8 00000080:00200000:0.0:1767316276.276799:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:0.0:1767316276.276945:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f130a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.276994:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.277156:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.277167:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316276.277281:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904569984:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316276.277413:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.282442:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904569984:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316276.282475:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.282502:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.282511:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.282522:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.282533:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.282545:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.282553:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.282575:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316276.282587:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f91033100 x1853163904569984/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/800 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316276.282599:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 216 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1767316276.282609:0:46893:0:(mdc_locks.c:969:mdc_finish_enqueue()) ### DoM lock is returned by: getattr, size: 0 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000002:00010000:0.0:1767316276.282619:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.282628:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1067 (0 0) 00000080:00200000:0.0:1767316276.282635:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1767316276.282700:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x42b:0x0](ffff9d1f88583a88): rc = 0 00000080:00200000:0.0:1767316276.282716:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x42b:0x0]: layout version change: 4294967294 -> 2 00000080:00010000:0.0:1767316276.282721:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42b:0x0](ffff9d1f88583a88) for lock 0xa4df836b164912fa 00000080:00010000:0.0:1767316276.282732:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42b:0x0](ffff9d1f88583a88) 00000080:00010000:0.0:1767316276.282736:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912fa from it ffffb2814198fb20 00010000:00010000:0.0:1767316276.282740:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.282748:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200008:0.0:1767316276.282798:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file f130a.replay-single:[0x200000bd1:0x42b:0x0](ffff9d1f88583a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1767316276.282805:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x42b:0x0](ffff9d1f88583a88),name=f130a.replay-single 00000002:00010000:0.0:1767316276.282824:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x42b:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.282833:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.282843:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316276.282850:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.282854:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42b:0x0](ffff9d1f88583a88) 00000080:00010000:0.0:1767316276.282858:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42b:0x0](ffff9d1f88583a88) for lock 0xa4df836b164912fa 00000080:00010000:0.0:1767316276.282862:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912fa from it ffffb2814198fcb8 00010000:00010000:0.0:1767316276.282865:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.282872:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316276.282889:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x42b:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1767316276.282899:0:46893:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x42b:0x0] 00000002:00010000:0.0:1767316276.282915:0:46893:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000002:00010000:0.0:1767316276.282921:0:46893:0:(mdc_dev.c:989:mdc_lock_init()) ### lock ffff9d1f8b3feb30, mdc lock ffff9d1f81322720, flags 400001002 00010000:00010000:0.0:1767316276.282931:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.282939:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000002:00010000:0.0:1767316276.282960:0:46893:0:(mdc_dev.c:448:mdc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=0 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00200000:0.0:1767316276.282971:0:46893:0:(cl_io.c:1802:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1767316276.282976:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 5/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.282983:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 4/0,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00200000:0.0:1767316276.282992:0:46893:0:(cl_io.c:1802:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1767316276.283087:0:46893:0:(file.c:1686:ll_merge_attr_nolock()) [0x200000bd1:0x42b:0x0] updating i_size 0 i_blocks 3 00000020:00010000:0.0:1767316276.283092:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d1f90487230 (ffff9d1f94298b48/0) at cl_lock_release():245 00000020:00010000:0.0:1767316276.283096:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d1f90487230 (ffff9d1f94298b48/0) at cl_lock_cancel():142 00000020:00010000:0.0:1767316276.283100:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d1f8b3feb30 (ffff9d1f94298638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1767316276.283104:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d1f90487230 (ffff9d1f94298b48/0) at cl_lock_fini():72 00000020:00010000:0.0:1767316276.283107:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d1f8b3feb30 (ffff9d1f94298b48/1) at cl_lock_fini():72 00000080:00200000:0.0:1767316276.283118:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x42b:0x0] ignore/verify layout 0/1, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1767316276.283158:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316276.283166:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.283295:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.283306:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.283313:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.283444:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316276.283447:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164912f3 00000080:00010000:0.0:1767316276.283451:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912f3 from it ffffb2814198fc10 00010000:00010000:0.0:1767316276.283455:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.283463:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.283472:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316276.283482:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f130a.replay-single, flags=0 00000080:00200000:0.0:1767316276.283487:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316276.283494:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.283500:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.283507:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.283513:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.283517:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316276.283519:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b164912f3 00000080:00010000:0.0:1767316276.283522:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b164912f3 from it ffffb2814198fd68 00010000:00010000:0.0:1767316276.283525:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.283532:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.283539:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 3 00000080:00200000:0.0:1767316276.283551:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348) 00010000:00010000:0.0:1767316276.283573:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.283594:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT gid 0 flags: 0x800028400000000 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316276.283608:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x42b:0x0] ignore/verify layout 1/0, layout version 2 restore needed 0 00000080:00200000:0.0:1767316276.283615:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x42b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1767316276.283622:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x42b:0x0]: losing layout lock 00010000:00010000:0.0:1767316276.283654:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x804829400000020 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.283660:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.283666:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1767316276.283669:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c85a00/0xa4df836b164912fa lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x804829400000020 nid: local remote: 0xcb3a17585d8ca06b expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.283899:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b164912f3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8ca04f expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000020:00100000:0.0:1767316276.283919:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1767316276.283931:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f91033100 x1853163904570240/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 504/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767316276.283956:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904570240:192.168.202.140@tcp:36:rm.0 00000100:00100000:0.0:1767316276.284168:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.289967:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904570240:192.168.202.140@tcp:36:rm.0 00000080:00200000:0.0:1767316276.290045:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x42b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1767316276.290058:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:0.0:1767316276.290075:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:0.0:1767316276.290094:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x42b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1767316276.290101:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x42b:0x0](ffff9d1f88583a88) 00000080:00200000:0.0:1767316276.290183:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316276.290199:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.290248:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.290257:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:0.0:1767316276.290267:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:0.0:1767316276.290285:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904570368:192.168.202.140@tcp:101:rm.0 00000100:00100000:0.0:1767316276.290356:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90a11300 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.297512:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904570368:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316276.297541:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.297553:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.297565:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.297575:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.297587:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.297595:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.297602:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.297611:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f91032680 x1853163904570368/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.297625:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.297638:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1767316276.297658:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316276.297922:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1767316276.297931:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491301 00000080:00010000:2.0:1767316276.297938:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316276.297942:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491301 from it ffffb2814198fb58 00010000:00010000:2.0:1767316276.297945:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.297953:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316276.297979:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.298155:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=8 00000080:00200000:2.0:1767316276.298160:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:2.0:1767316276.298315:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f131a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.298456:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.298465:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.298474:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316276.298499:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904570496:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316276.298644:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759300 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.304213:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904570496:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316276.304239:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.304282:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.304290:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.304301:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.304312:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.304334:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.304342:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.304349:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.304360:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f967cea00 x1853163904570496/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/800 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.304370:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 216 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1767316276.304380:0:46893:0:(mdc_locks.c:969:mdc_finish_enqueue()) ### DoM lock is returned by: getattr, size: 8 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000002:00010000:2.0:1767316276.304410:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.304422:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1071 (0 0) 00000080:00200000:2.0:1767316276.304429:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1767316276.304477:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x42f:0x0](ffff9d1f885d2a08): rc = 0 00000080:00200000:2.0:1767316276.304494:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x42f:0x0]: layout version change: 4294967294 -> 2 00000080:00010000:2.0:1767316276.304499:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42f:0x0](ffff9d1f885d2a08) for lock 0xa4df836b16491308 00000080:00010000:2.0:1767316276.304510:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42f:0x0](ffff9d1f885d2a08) 00000080:00010000:2.0:1767316276.304514:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491308 from it ffffb2814198fb20 00010000:00010000:2.0:1767316276.304517:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.304525:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200008:2.0:1767316276.304563:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file f131a.replay-single:[0x200000bd1:0x42f:0x0](ffff9d1f885d2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316276.304570:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x42f:0x0](ffff9d1f885d2a08),name=f131a.replay-single 00000002:00010000:2.0:1767316276.304589:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x42f:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.304596:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.304606:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.304612:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316276.304615:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42f:0x0](ffff9d1f885d2a08) 00000080:00010000:2.0:1767316276.304625:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42f:0x0](ffff9d1f885d2a08) for lock 0xa4df836b16491308 00000080:00010000:2.0:1767316276.304629:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491308 from it ffffb2814198fcb8 00010000:00010000:2.0:1767316276.304632:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.304639:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.304656:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x42f:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1767316276.304666:0:46893:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x42f:0x0] 00000002:00010000:2.0:1767316276.304707:0:46893:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000002:00010000:2.0:1767316276.304714:0:46893:0:(mdc_dev.c:989:mdc_lock_init()) ### lock ffff9d1f96775bb0, mdc lock ffff9d1f9941a4c0, flags 400001002 00010000:00010000:2.0:1767316276.304730:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.304739:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000002:00010000:2.0:1767316276.304750:0:46893:0:(mdc_dev.c:448:mdc_lock_lvb_update()) ### acquired size 8, setting rss=8; kms=8, end=0 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00200000:2.0:1767316276.304760:0:46893:0:(cl_io.c:1802:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1767316276.304764:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 5/1,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.304771:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 4/0,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x800020000000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000020:00200000:2.0:1767316276.304778:0:46893:0:(cl_io.c:1802:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1767316276.304787:0:46893:0:(file.c:1686:ll_merge_attr_nolock()) [0x200000bd1:0x42f:0x0] updating i_size 8 i_blocks 11 00000020:00010000:2.0:1767316276.304797:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d1f875aeb58 (ffff9d1f88d7e3f8/0) at cl_lock_release():245 00000020:00010000:2.0:1767316276.304800:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d1f875aeb58 (ffff9d1f88d7e3f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1767316276.304803:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d1f96775bb0 (ffff9d1f88d7e638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1767316276.304808:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d1f875aeb58 (ffff9d1f88d7e3f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1767316276.304810:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d1f96775bb0 (ffff9d1f88d7e3f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1767316276.304819:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x42f:0x0] ignore/verify layout 0/1, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.304864:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316276.304873:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.304879:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.304886:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316276.304893:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316276.304897:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316276.304899:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491301 00000080:00010000:2.0:1767316276.304903:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491301 from it ffffb2814198fc10 00010000:00010000:2.0:1767316276.304906:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.304913:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316276.304921:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.304929:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f131a.replay-single, flags=0 00000080:00200000:2.0:1767316276.304933:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316276.304939:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.304944:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.304958:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316276.304965:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316276.304969:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316276.304971:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491301 00000080:00010000:2.0:1767316276.304975:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491301 from it ffffb2814198fd68 00010000:00010000:2.0:1767316276.304978:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.304984:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316276.304992:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 3 00000080:00200000:2.0:1767316276.304997:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=f131a.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348) 00010000:00010000:2.0:1767316276.305016:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.305037:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 3 type: IBT gid 0 flags: 0x800028400000000 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.305051:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x42f:0x0] ignore/verify layout 1/0, layout version 2 restore needed 0 00000080:00200000:2.0:1767316276.305058:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x42f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.305064:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x42f:0x0]: losing layout lock 00010000:00010000:2.0:1767316276.305093:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x804829400000020 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.305101:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.305112:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1767316276.305115:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b16491308 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x42f:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x804829400000020 nid: local remote: 0xcb3a17585d8ca0b1 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.305133:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93c84000/0xa4df836b16491301 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8ca095 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000020:00100000:2.0:1767316276.305151:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316276.305158:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f967cea00 x1853163904570752/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 504/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316276.305178:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904570752:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316276.305427:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759300 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.312022:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904570752:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316276.312093:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x42f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1767316276.312107:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1767316276.312116:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:2.0:1767316276.312464:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x42f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.312474:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x42f:0x0](ffff9d1f885d2a08) 00000080:00200000:2.0:1767316276.312583:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316276.312605:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.312664:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.312674:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.313514:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316276.313557:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cca80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904570880:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316276.313644:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759300 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.316648:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cca80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904570880:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316276.316673:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.316772:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.316783:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.316792:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.316803:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.316811:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.316818:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.316829:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f967cca80 x1853163904570880/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.316842:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.316868:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1767316276.316875:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316276.316899:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1767316276.316904:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b1649130f 00000080:00010000:2.0:1767316276.316910:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316276.316913:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649130f from it ffffb2814198fb58 00010000:00010000:2.0:1767316276.316916:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.316925:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316276.316952:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.316974:0:46893:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), flags=8 00000080:00200000:2.0:1767316276.316979:0:46893:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348), intent=getattr 00000002:00010000:2.0:1767316276.317146:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f132a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.317443:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.317451:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:2.0:1767316276.317461:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:2.0:1767316276.317484:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cca80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904571008:192.168.202.140@tcp:101:rm.0 00000100:00100000:2.0:1767316276.317633:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759300 going to sleep for 11 seconds 00000100:00100000:0.0:1767316276.327622:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cca80 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904571008:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316276.327650:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.327674:0:46893:0:(ldlm_request.c:717:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.327706:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.327717:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.327745:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.327758:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.327766:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.327773:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1767316276.327783:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f967cca80 x1853163904571008/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 592/848 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1767316276.327796:0:46893:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 264 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00010000:0.0:1767316276.327808:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.327818:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589937617/1074 (0 0) 00000080:00200000:0.0:1767316276.327825:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1767316276.327874:0:46893:0:(namei.c:151:ll_iget()) got inode: [0x200000bd1:0x432:0x0](ffff9d1f885821c8): rc = 0 00000080:00200000:0.0:1767316276.327889:0:46893:0:(file.c:6809:ll_layout_conf()) [0x200000bd1:0x432:0x0]: layout version change: 4294967294 -> 4 00000080:00010000:0.0:1767316276.327895:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x432:0x0](ffff9d1f885821c8) for lock 0xa4df836b16491316 00000080:00010000:0.0:1767316276.327904:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x432:0x0](ffff9d1f885821c8) 00000080:00010000:0.0:1767316276.327908:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491316 from it ffffb2814198fb20 00010000:00010000:0.0:1767316276.327918:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.327926:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200008:0.0:1767316276.327964:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file f132a.replay-single:[0x200000bd1:0x432:0x0](ffff9d1f885821c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1767316276.327972:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x432:0x0](ffff9d1f885821c8),name=f132a.replay-single 00000002:00010000:0.0:1767316276.327993:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x432:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.328012:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.328022:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316276.328028:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.328033:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x432:0x0](ffff9d1f885821c8) 00000080:00010000:0.0:1767316276.328036:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x432:0x0](ffff9d1f885821c8) for lock 0xa4df836b16491316 00000080:00010000:0.0:1767316276.328040:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491316 from it ffffb2814198fcb8 00010000:00010000:0.0:1767316276.328043:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:0.0:1767316276.328050:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316276.328067:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x432:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1767316276.328077:0:46893:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x432:0x0] 00000008:00010000:0.0:1767316276.328095:0:46893:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1767316276.328098:0:46893:0:(osc_lock.c:1248:osc_lock_init()) ### lock ffff9d1f918a1430, osc lock ffff9d1f81322720, flags 0x400001002 00000008:00010000:0.0:1767316276.328135:0:46893:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1767316276.328137:0:46893:0:(osc_lock.c:1248:osc_lock_init()) ### lock ffff9d1f918a1470, osc lock ffff9d1f81322850, flags 0x400001002 00000008:00010000:0.0:1767316276.328149:0:46893:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1767316276.328151:0:46893:0:(osc_lock.c:1248:osc_lock_init()) ### lock ffff9d1f918a14b0, osc lock ffff9d1f813225f0, flags 0x400001002 00010000:00010000:0.0:1767316276.328206:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1767316276.328214:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->1048575] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1767316276.328262:0:2490:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1767316276.328277:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904571136:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316276.328485:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1767316276.328494:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [1048576->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1767316276.328523:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904571264:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316276.328571:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1767316276.328579:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000020:00200000:0.0:1767316276.328593:0:46893:0:(cl_io.c:1802:__cl_sync_io_note()) Noting completion of 1 items, 3 items remaining. 00000100:00100000:0.0:1767316276.328601:0:2490:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1767316276.328607:0:2490:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904571392:192.168.202.140@tcp:101:rm.0 00010000:00010000:1.0:1767316276.335406:0:2490:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->1048575] (req 0->1048575) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316276.335433:0:2490:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316276.335448:0:2490:0:(ldlm_resource.c:1525:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316276.335475:0:2490:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000008:00010000:1.0:1767316276.335490:0:2490:0:(osc_lock.c:179:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000020:00200000:1.0:1767316276.335501:0:2490:0:(cl_io.c:1802:__cl_sync_io_note()) Noting completion of 1 items, 2 items remaining. 00010000:00010000:1.0:1767316276.335505:0:2490:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316276.335513:0:2490:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316276.335520:0:2490:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316276.335528:0:2490:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000100:00100000:1.0:1767316276.335547:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904571136:192.168.202.140@tcp:101:rm.0 00010000:00010000:0.0:1767316276.338863:0:2490:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1767316276.338890:0:2490:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1767316276.338906:0:2490:0:(ldlm_resource.c:1525:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1767316276.338918:0:2490:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000008:00010000:0.0:1767316276.338933:0:2490:0:(osc_lock.c:179:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000020:00200000:0.0:1767316276.338945:0:2490:0:(cl_io.c:1802:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1767316276.338949:0:2490:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1767316276.338958:0:2490:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1767316276.338965:0:2490:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1767316276.338973:0:2490:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000100:00100000:0.0:1767316276.339024:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904571392:192.168.202.140@tcp:101:rm.0 00010000:00010000:1.0:1767316276.341602:0:2490:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [1048576->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316276.341630:0:2490:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316276.341663:0:2490:0:(ldlm_resource.c:1525:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316276.341675:0:2490:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000008:00010000:1.0:1767316276.341732:0:2490:0:(osc_lock.c:179:osc_lock_lvb_update()) ### acquired size 2097152, setting rss=2097152; kms=2097152, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000020:00200000:1.0:1767316276.341744:0:2490:0:(cl_io.c:1802:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1767316276.341752:0:2490:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316276.341760:0:2490:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316276.341767:0:2490:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316276.341776:0:2490:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000100:00100000:1.0:1767316276.341794:0:2490:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2490:1853163904571264:192.168.202.140@tcp:101:rm.0 00000080:00200000:0.0:1767316276.341875:0:46893:0:(file.c:1686:ll_merge_attr_nolock()) [0x200000bd1:0x432:0x0] updating i_size 2097152 i_blocks 2097 00000020:00010000:0.0:1767316276.341881:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d1f90487230 (ffff9d1f94298b48/0) at cl_lock_release():245 00000020:00010000:0.0:1767316276.341884:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d1f90487230 (ffff9d1f94298b48/0) at cl_lock_cancel():142 00000020:00010000:0.0:1767316276.341889:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d1f918a1430 (ffff9d1f94298638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1767316276.341894:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d1f918a1470 (ffff9d1f942981b8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1767316276.341898:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d1f918a14b0 (ffff9d1f942985a8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1767316276.341900:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d1f90487230 (ffff9d1f94298b48/0) at cl_lock_fini():72 00000020:00010000:0.0:1767316276.341903:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d1f918a1430 (ffff9d1f94298b48/1) at cl_lock_fini():72 00000020:00010000:0.0:1767316276.341909:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d1f918a1470 (ffff9d1f94298b48/1) at cl_lock_fini():72 00000020:00010000:0.0:1767316276.341911:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d1f918a14b0 (ffff9d1f94298b48/1) at cl_lock_fini():72 00000080:00200000:0.0:1767316276.341927:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x432:0x0] ignore/verify layout 0/1, layout version 4 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1767316276.341975:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316276.342141:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.342156:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.342168:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.342193:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.342200:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316276.342204:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b1649130f 00000080:00010000:0.0:1767316276.342209:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649130f from it ffffb2814198fc10 00010000:00010000:0.0:1767316276.342213:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.342226:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.342236:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:0.0:1767316276.342245:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f132a.replay-single, flags=0 00000080:00200000:0.0:1767316276.342251:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:0.0:1767316276.342258:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767316276.342263:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.342271:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.342277:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767316276.342282:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:0.0:1767316276.342284:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b1649130f 00000080:00010000:0.0:1767316276.342288:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b1649130f from it ffffb2814198fd68 00010000:00010000:0.0:1767316276.342291:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.342297:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767316276.342305:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 3 00000080:00200000:0.0:1767316276.342310:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=f132a.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348) 00010000:00010000:0.0:1767316276.342664:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767316276.342860:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00000080:00200000:0.0:1767316276.342871:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd1:0x432:0x0]: losing layout lock 00010000:00010000:1.0:1767316276.343587:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1767316276.343605:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767316276.343612:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1767316276.343615:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2e00/0xa4df836b16491316 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x432:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8ca0f7 expref: -99 pid: 46893 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1767316276.343634:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f909f2400/0xa4df836b1649130f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8ca0db expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000020:00100000:1.0:1767316276.343655:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1767316276.343670:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f918d2a00 x1853163904571648/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 504/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767316276.343783:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2a00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904571648:192.168.202.140@tcp:36:rm.0 00000100:00100000:1.0:1767316276.343878:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f874c5f00 going to sleep for 11 seconds 00000100:00100000:1.0:1767316276.349558:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2a00 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904571648:192.168.202.140@tcp:36:rm.0 00000080:00200000:1.0:1767316276.349621:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd1:0x432:0x0] ignore/verify layout 1/0, layout version 4 restore needed 0 00020000:00200000:1.0:1767316276.349637:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:1.0:1767316276.349722:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:1.0:1767316276.349784:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd1:0x432:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1767316276.349845:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x432:0x0](ffff9d1f885821c8) 00000008:00010000:1.0:1767316276.349890:0:46893:0:(osc_object.c:209:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x502:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000008:00010000:1.0:1767316276.349918:0:46893:0:(osc_object.c:209:osc_object_ast_clear()) ### update lock size 2097152 blocks 2093 [cma]time: 1767315895 1767315895 0 by oinfo size 2097152 blocks 2093 [cma]time 1767315895 1767315895 0 ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x502:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000008:00010000:1.0:1767316276.349959:0:46893:0:(osc_object.c:209:osc_object_ast_clear()) ### update lock size 0 blocks 3 [cma]time: 1767315894 1767315894 0 by oinfo size 0 blocks 3 [cma]time 1767315894 1767315894 0 ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x503:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1767316276.350243:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316276.350261:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316276.350314:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 46893 initiator: MDT0 00010000:00010000:1.0:1767316276.350412:0:46893:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00010000:00010000:1.0:1767316276.350423:0:46893:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 46893 initiator: MDT0 00000100:00100000:1.0:1767316276.350446:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904571776:192.168.202.140@tcp:101:rm.0 00000100:00100000:1.0:1767316276.350597:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f874c5f00 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.356956:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904571776:192.168.202.140@tcp:101:rm.0 00010000:00010000:2.0:1767316276.357155:0:46893:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.357167:0:46893:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.357299:0:46893:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.357315:0:46893:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.357417:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.357426:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.357434:0:46893:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767316276.357446:0:46893:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d1f918d2300 x1853163904571776/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/688 e 0 to 0 dl 1767316292 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767316276.357459:0:46893:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.357472:0:46893:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9d1f90e0ac00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1767316276.357479:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767316276.357509:0:46893:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 10752 00000080:00010000:2.0:1767316276.357524:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491332 00000080:00010000:2.0:1767316276.357530:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316276.357533:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491332 from it ffffb2814198fb58 00010000:00010000:2.0:1767316276.357536:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.357544:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316276.357570:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.357583:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f203.replay-single, flags=8 00000080:00200008:2.0:1767316276.357599:0:46893:0:(file.c:6252:ll_getattr_dentry()) START file f203.replay-single:[0x2000059f1:0x1:0x0](ffff9d1f885d7448), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1767316276.357606:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x2000059f1:0x1:0x0](ffff9d1f885d7448),name=f203.replay-single 00000002:00010000:2.0:1767316276.357626:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x2000059f1:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.357635:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 2/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.357645:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 2/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.357651:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316276.357655:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000059f1:0x1:0x0](ffff9d1f885d7448) 00000080:00010000:2.0:1767316276.357657:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x2000059f1:0x1:0x0](ffff9d1f885d7448) for lock 0xa4df836b16491148 00000080:00010000:2.0:1767316276.357660:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491148 from it ffffb2814198fcb8 00010000:00010000:2.0:1767316276.357663:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 3/1,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.357669:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 2/0,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.358171:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x2000059f1:0x1:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1767316276.358421:0:46893:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x2000059f1:0x1:0x0] 00000008:00010000:2.0:1767316276.358452:0:46893:0:(lustre_osc.h:163:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1767316276.358456:0:46893:0:(osc_lock.c:1248:osc_lock_init()) ### lock ffff9d1f96775bb0, osc lock ffff9d1f9941a4c0, flags 0x400001002 00010000:00010000:2.0:1767316276.358475:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316276.358485:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 1048575) ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00000020:00200000:2.0:1767316276.358497:0:46893:0:(cl_io.c:1802:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1767316276.358501:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x810020000000000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316276.358507:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00000020:00200000:2.0:1767316276.358518:0:46893:0:(cl_io.c:1802:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1767316276.358528:0:46893:0:(file.c:1686:ll_merge_attr_nolock()) [0x2000059f1:0x1:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1767316276.358533:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d1f875aeb58 (ffff9d1f88d7e3f8/0) at cl_lock_release():245 00000020:00010000:2.0:1767316276.358536:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d1f875aeb58 (ffff9d1f88d7e3f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1767316276.358539:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d1f96775bb0 (ffff9d1f88d7e638/1) at cl_lock_cancel():142 00000020:00010000:2.0:1767316276.358543:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d1f875aeb58 (ffff9d1f88d7e3f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1767316276.358546:0:46893:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d1f96775bb0 (ffff9d1f88d7e3f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1767316276.358557:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x2000059f1:0x1:0x0] ignore/verify layout 0/1, layout version 12 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.358592:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316276.358603:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.358618:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.358626:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316276.358632:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316276.358638:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316276.358641:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491332 00000080:00010000:2.0:1767316276.358646:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491332 from it ffffb2814198fc10 00010000:00010000:2.0:1767316276.358649:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.358656:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316276.358664:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 1 00000080:00200000:2.0:1767316276.360446:0:46893:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f203.replay-single, flags=0 00000080:00200000:2.0:1767316276.360465:0:46893:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:2.0:1767316276.360493:0:46893:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767316276.360508:0:46893:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.360523:0:46893:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316276.360530:0:46893:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1767316276.360539:0:46893:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) 00000080:00010000:2.0:1767316276.360543:0:46893:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d1f90125348) for lock 0xa4df836b16491332 00000080:00010000:2.0:1767316276.360549:0:46893:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xa4df836b16491332 from it ffffb2814198fd68 00010000:00010000:2.0:1767316276.360566:0:46893:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.360575:0:46893:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767316276.360586:0:46893:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348), inode mode 41ed mask 3 00000080:00200000:2.0:1767316276.360593:0:46893:0:(namei.c:2354:ll_unlink()) VFS Op:name=f203.replay-single, dir=[0x200000007:0x1:0x0](ffff9d1f90125348) 00010000:00010000:2.0:1767316276.360620:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.360642:0:46893:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 2/0,0 mode: PR/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1767316276.360651:0:46893:0:(vvp_object.c:114:vvp_conf_set()) [0x2000059f1:0x1:0x0]: losing layout lock 00010000:00010000:2.0:1767316276.360875:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 1/0,0 mode: --/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.360884:0:46893:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767316276.360890:0:46893:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1767316276.360893:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f8b385c00/0xa4df836b16491148 lrc: 0/0,0 mode: --/PR res: [0x2000059f1:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xcb3a17585d8c97e3 expref: -99 pid: 46154 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767316276.360918:0:46893:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccd000/0xa4df836b16491332 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xcb3a17585d8ca136 expref: -99 pid: 46893 timeout: 0 lvb_type: 0 00000020:00100000:2.0:1767316276.360937:0:46893:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d1f98126000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1767316276.360950:0:46893:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d1f967cd500 x1853163904572032/t0(0) o36->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 504/1800 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316276.360988:0:46893:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cd500 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904572032:192.168.202.140@tcp:36:rm.0 00000100:00100000:2.0:1767316276.361475:0:46893:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f96759240 going to sleep for 11 seconds 00000100:00100000:2.0:1767316276.372856:0:46893:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cd500 pname:cluuid:pid:xid:nid:opc:job rm:45d767c3-7963-411a-be8f-bcbeb1c06075:46893:1853163904572032:192.168.202.140@tcp:36:rm.0 00000080:00200000:2.0:1767316276.372912:0:46893:0:(vvp_io.c:1854:vvp_io_init()) [0x2000059f1:0x1:0x0] ignore/verify layout 1/0, layout version 12 restore needed 0 00020000:00200000:2.0:1767316276.372925:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1767316276.372935:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[1] flags 0x0 00020000:00200000:2.0:1767316276.372936:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[2] flags 0x0 00020000:00200000:2.0:1767316276.372938:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[3] flags 0x0 00020000:00200000:2.0:1767316276.372939:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[4] flags 0x0 00020000:00200000:2.0:1767316276.372941:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[5] flags 0x0 00020000:00200000:2.0:1767316276.372942:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[6] flags 0x0 00020000:00200000:2.0:1767316276.372944:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[7] flags 0x0 00020000:00200000:2.0:1767316276.372945:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[8] flags 0x0 00020000:00200000:2.0:1767316276.372946:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[9] flags 0x0 00020000:00200000:2.0:1767316276.372948:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[10] flags 0x0 00020000:00200000:2.0:1767316276.372949:0:46893:0:(lov_io.c:860:lov_io_iter_init()) component[11] flags 0x0 00000080:00200000:2.0:1767316276.372967:0:46893:0:(vvp_io.c:289:vvp_io_fini()) [0x2000059f1:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1767316276.372974:0:46893:0:(llite_lib.c:1953:ll_clear_inode()) VFS Op:inode=[0x2000059f1:0x1:0x0](ffff9d1f885d7448) 00000008:00010000:2.0:1767316276.373010:0:46893:0:(osc_object.c:209:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x800020000000000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00000100:00100000:2.0:1767316276.534949:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904572288:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316276.535060:0:2492:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1767316276.535066:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2492:1853163904572160:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316276.535090:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904572416:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316276.541407:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904572416:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316276.552443:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904572288:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316276.552495:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2492:1853163904572160:192.168.202.140@tcp:400:kworker.0 00010000:00010000:0.0:1767316277.175888:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316278.198955:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316279.223337:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1767316279.500288:0:47161:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: === replay-single: finish cleanup 20:11:17 (1767316277) === 00010000:00010000:3.0:1767316280.247784:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767316281.270938:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316281.655479:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904572672:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316281.655621:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904572928:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316281.655654:0:2493:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1767316281.655660:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904572544:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316281.655708:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904572800:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316281.659591:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904572544:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316281.659618:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904572800:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316281.659630:0:2493:0:(client.c:3081:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d1f98126000: committing for last_committed 51539607620 gen 2 00000100:00100000:2.0:1767316281.659904:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904572672:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316281.659918:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904572928:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316281.820619:0:2483:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.140@tcp (source: 12345-192.168.202.140@tcp) 00000100:00100000:0.0:1767316281.820779:0:13861:0:(service.c:2279:ptlrpc_server_handle_req_in()) unwrap req x1853163946297344 00000100:00100000:0.0:1767316281.820792:0:13861:0:(service.c:2344:ptlrpc_server_handle_req_in()) got req x1853163946297344 00000100:00100000:0.0:1767316281.820856:0:13861:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.140@tcp, seq: 1433 00000100:00100000:0.0:1767316281.820865:0:13861:0:(service.c:2530:ptlrpc_server_handle_request()) Handling RPC req@ffff9d1f918d2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:51762:x1853163946297344:12345-192.168.202.140@tcp:104: 00010000:00010000:0.0:1767316281.820885:0:13861:0:(ldlm_lockd.c:2505:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x420000010000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:0.0:1767316281.820983:0:13861:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1767316281.820989:0:13861:0:(service.c:2588:ptlrpc_server_handle_request()) Handled RPC req@ffff9d1f918d2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:51762:x1853163946297344:12345-192.168.202.140@tcp:104: Request processed in 125us (383us total) trans 0 rc 0/0 00000100:00100000:0.0:1767316281.820996:0:13861:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.140@tcp, seq: 1433 00010000:00010000:2.0:1767316281.821311:0:15045:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1767316281.821318:0:15045:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x420000010000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.821327:0:15045:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f8b384e00 already unused, calling callback (ffffffffc0fbe0e0) 00010000:00010000:2.0:1767316281.821333:0:15045:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x428400010000 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.821391:0:15045:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 2/0,0 mode: --/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x4c29400010020 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.821413:0:15045:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316281.821446:0:15045:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 1/0,0 mode: --/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x4c29400010020 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.821454:0:15045:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f8b384e00/0xa4df836b1649116b lrc: 0/0,0 mode: --/PR res: [0x240000400:0x562:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x4c29400010020 nid: local remote: 0xcb3a17585d8c97ea expref: -99 pid: 46154 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.821476:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316281.821478:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316281.821480:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316281.821489:0:2493:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1767316281.821503:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904573056:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1767316281.838246:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904573056:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1767316281.848301:0:2483:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.140@tcp (source: 12345-192.168.202.140@tcp) 00000100:00100000:1.0:1767316281.848648:0:7197:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1767316281.848673:0:7197:0:(service.c:2279:ptlrpc_server_handle_req_in()) unwrap req x1853163946298112 00000100:00100000:1.0:1767316281.848721:0:7197:0:(service.c:2344:ptlrpc_server_handle_req_in()) got req x1853163946298112 00000100:00100000:1.0:1767316281.848783:0:7197:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.140@tcp, seq: 1434 00000100:00100000:1.0:1767316281.848793:0:7197:0:(service.c:2530:ptlrpc_server_handle_request()) Handling RPC req@ffff9d1f89213850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:52645:x1853163946298112:12345-192.168.202.140@tcp:104: 00010000:00010000:1.0:1767316281.848811:0:7197:0:(ldlm_lockd.c:2505:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316281.848923:0:7197:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1767316281.848931:0:7197:0:(service.c:2588:ptlrpc_server_handle_request()) Handled RPC req@ffff9d1f89213850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:52645:x1853163946298112:12345-192.168.202.140@tcp:104: Request processed in 139us (642us total) trans 0 rc 0/0 00000100:00100000:1.0:1767316281.848938:0:7197:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.140@tcp, seq: 1434 00010000:00010000:2.0:1767316281.849127:0:15046:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1767316281.849135:0:15046:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.849144:0:15046:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f93c84a00 already unused, calling callback (ffffffffc0fbe0e0) 00010000:00010000:2.0:1767316281.849151:0:15046:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.849198:0:15046:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.849206:0:15046:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316281.849245:0:15046:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.849252:0:15046:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d1f98126000 lock: ffff9d1f93c84a00/0xa4df836b16491324 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0xcb3a17585d8ca105 expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.849274:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316281.849275:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316281.849277:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767316281.849324:0:2492:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1767316281.849349:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904573184:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1767316281.865272:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904573184:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1767316281.889386:0:2483:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.140@tcp (source: 12345-192.168.202.140@tcp) 00000100:00100000:1.0:1767316281.889484:0:2483:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.140@tcp (source: 12345-192.168.202.140@tcp) 00000100:00100000:1.0:1767316281.889518:0:7197:0:(service.c:2279:ptlrpc_server_handle_req_in()) unwrap req x1853163946299264 00000100:00100000:1.0:1767316281.889528:0:7197:0:(service.c:2344:ptlrpc_server_handle_req_in()) got req x1853163946299264 00000100:00100000:0.0:1767316281.889559:0:13861:0:(service.c:2279:ptlrpc_server_handle_req_in()) unwrap req x1853163946299392 00000100:00100000:0.0:1767316281.889563:0:13861:0:(service.c:2344:ptlrpc_server_handle_req_in()) got req x1853163946299392 00000100:00100000:1.0:1767316281.889577:0:7197:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.140@tcp, seq: 1435 00000100:00100000:1.0:1767316281.889586:0:7197:0:(service.c:2530:ptlrpc_server_handle_request()) Handling RPC req@ffff9d1f918d2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+5:51762:x1853163946299264:12345-192.168.202.140@tcp:104: 00000100:00100000:0.0:1767316281.889588:0:13861:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.140@tcp, seq: 1436 00000100:00100000:0.0:1767316281.889593:0:13861:0:(service.c:2530:ptlrpc_server_handle_request()) Handling RPC req@ffff9d1f918d2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+5:47927:x1853163946299392:12345-192.168.202.140@tcp:104: 00010000:00010000:0.0:1767316281.889605:0:13861:0:(ldlm_lockd.c:2505:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x420000010000 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316281.889605:0:7197:0:(ldlm_lockd.c:2505:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:1.0:1767316281.889721:0:7197:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1767316281.889728:0:7197:0:(service.c:2588:ptlrpc_server_handle_request()) Handled RPC req@ffff9d1f918d2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+5:51762:x1853163946299264:12345-192.168.202.140@tcp:104: Request processed in 143us (359us total) trans 0 rc 0/0 00010000:00010000:0.0:1767316281.889732:0:13861:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:1.0:1767316281.889736:0:7197:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.140@tcp, seq: 1435 00000100:00100000:0.0:1767316281.889751:0:13861:0:(service.c:2588:ptlrpc_server_handle_request()) Handled RPC req@ffff9d1f918d2a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:47927:x1853163946299392:12345-192.168.202.140@tcp:104: Request processed in 158us (268us total) trans 0 rc 0/0 00000100:00100000:0.0:1767316281.889756:0:13861:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.140@tcp, seq: 1436 00010000:00010000:2.0:1767316281.889841:0:7208:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1767316281.889850:0:7208:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.889858:0:7208:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f93c84200 already unused, calling callback (ffffffffc0fbe0e0) 00010000:00010000:2.0:1767316281.889864:0:7208:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.889910:0:7208:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 2/0,0 mode: --/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.889917:0:7208:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316281.889939:0:7208:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 1/0,0 mode: --/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.889946:0:7208:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c84200/0xa4df836b1649132b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x503:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0xcb3a17585d8ca10c expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.889970:0:7208:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1767316281.889980:0:7208:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x420000010000 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.889986:0:7208:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9d1f93c85400 already unused, calling callback (ffffffffc0fbe0e0) 00010000:00010000:2.0:1767316281.889990:0:7208:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x428400010000 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.890007:0:7208:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 2/0,0 mode: --/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x4c29400010020 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.890012:0:7208:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1767316281.890018:0:7208:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 1/0,0 mode: --/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x4c29400010020 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.890025:0:7208:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d1f98126000 lock: ffff9d1f93c85400/0xa4df836b1649131d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x502:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x4c29400010020 nid: local remote: 0xcb3a17585d8ca0fe expref: -99 pid: 46893 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1767316281.890037:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316281.890038:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316281.890040:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316281.890056:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904573312:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1767316281.890132:0:2492:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1767316281.890139:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904573440:192.168.202.140@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1767316281.890168:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316281.893475:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904573440:192.168.202.140@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1767316281.893492:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904573312:192.168.202.140@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1767316282.295118:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316283.318856:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316284.343147:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316285.367258:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316286.394537:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767316286.780198:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904573696:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316286.780312:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904573952:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316286.780350:0:2493:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1767316286.780357:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904573568:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316286.780386:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904573824:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316286.797863:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904573824:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316286.797884:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904573952:192.168.202.140@tcp:400:kworker.0 00010000:00010000:2.0:1767316287.414853:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316288.440139:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316289.463847:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316290.487500:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316291.511983:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316291.895021:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904574208:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316291.895138:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904574464:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316291.895171:0:2491:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1767316291.895188:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2491:1853163904574080:192.168.202.140@tcp:400:kworker.0 00000100:00100000:1.0:1767316291.895217:0:2491:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904574336:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316291.898121:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904574464:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316291.898479:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904574336:192.168.202.140@tcp:400:kworker.0 00010000:00010000:2.0:1767316292.535617:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316293.558928:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316294.585056:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316295.609644:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316296.630846:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316297.015080:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904574720:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316297.015191:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2492:1853163904574592:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316297.017332:0:2493:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904574976:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316297.018935:0:2492:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904574848:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316297.025711:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:45d767c3-7963-411a-be8f-bcbeb1c06075:2492:1853163904574848:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316297.027292:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904574976:192.168.202.140@tcp:400:kworker.0 00010000:00010000:3.0:1767316297.660527:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316298.678869:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316299.702848:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316300.726848:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316301.751724:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1767316302.135043:0:2493:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316286/real 1767316286] req@ffff9d1f92f70380 x1853163904573568/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1767316302 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316302.135083:0:2493:0:(client.c:2522:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d1f92f70380 x1853163904573568/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1767316302 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316302.135093:0:2493:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1767316302.135098:0:2493:0:(import.c:185:ptlrpc_set_import_discon()) MGC192.168.202.140@tcp: Connection to MGS (at 192.168.202.140@tcp) was lost; in progress operations using this service will fail 00000100:00080000:2.0:1767316302.135101:0:2493:0:(import.c:191:ptlrpc_set_import_discon()) import MGS@192.168.202.140@tcp for MGC192.168.202.140@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1767316302.135104:0:2493:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1767316302.135108:0:2493:0:(client.c:3559:ptlrpc_abort_inflight()) @@@ inflight req@ffff9d1f92f70380 x1853163904573568/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 1 dl 1767316302 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316302.135137:0:2493:0:(client.c:3559:ptlrpc_abort_inflight()) @@@ inflight req@ffff9d1f918d2300 x1853163904574080/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 0 dl 1767316307 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316302.135173:0:2493:0:(client.c:3559:ptlrpc_abort_inflight()) @@@ inflight req@ffff9d1f967cdf80 x1853163904574592/t0(0) o400->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 224/224 e 0 to 0 dl 1767316313 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:2.0:1767316302.135194:0:2493:0:(mgc_request.c:1165:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1767316302.135197:0:2493:0:(pinger.c:192:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1767316302.135199:0:2493:0:(mgc_request.c:1165:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1767316302.135201:0:2493:0:(import.c:416:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00000400:2.0:1767316302.135211:0:2493:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316286/real 1767316286] req@ffff9d1f92f70e00 x1853163904573696/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316302 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316302.135227:0:2493:0:(client.c:2522:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d1f92f70e00 x1853163904573696/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316302 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316302.135234:0:2493:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1767316302.135238:0:2493:0:(import.c:181:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff9d1f98126000: Connection to lustre-MDT0000 (at 192.168.202.140@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1767316302.135242:0:2493:0:(import.c:416:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00100000:2.0:1767316302.135257:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2493:1853163904573568:192.168.202.140@tcp:400:kworker.0 00000100:00100000:2.0:1767316302.135265:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904573696:192.168.202.140@tcp:400:kworker.0 00000100:00100000:3.0:1767316302.135367:0:2492:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2492:1853163904574592:192.168.202.140@tcp:400:kworker.0 00000100:00100000:0.0:1767316302.135451:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2491:1853163904574080:192.168.202.140@tcp:400:kworker.0 00000100:00080000:1.0:1767316302.135950:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316302.135965:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767316302.135968:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316302.135974:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1479 00000100:00080000:1.0:1767316302.136037:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767316302.136073:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0700 x1853163904575104/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316302.136103:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316302.136107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767316302.136109:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316302.136112:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1494 00000100:00080000:1.0:1767316302.136118:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767316302.136128:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767316302.136139:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1c00 x1853163904575232/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316302.136166:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad95800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:02000000:1.0:1767316302.136169:0:36:0:(import.c:2080:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff9d1f98126000: disconnect after 21s idle 00000100:00100000:2.0:1767316302.136970:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904575104:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767316302.137081:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904575232:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767316302.147882:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad93000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:02000000:1.0:1767316302.147889:0:36:0:(import.c:2080:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff9d1f98126000: disconnect after 21s idle 00000100:00100000:0.0:1767316302.149497:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904575360:192.168.202.140@tcp:9:kworker.0 00000100:00100000:0.0:1767316302.149599:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904575488:192.168.202.140@tcp:9:kworker.0 00000100:00080000:2.0:1767316302.164500:0:2489:0:(import.c:1990:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff9d1f918d1500 x1853163904575360/t0(0) o9->lustre-OST0000-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316312 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316302.164516:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad95800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1767316302.164536:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904575360:192.168.202.140@tcp:9:kworker.0 00000100:00080000:2.0:1767316302.164553:0:2489:0:(import.c:1990:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff9d1f918d1180 x1853163904575488/t0(0) o9->lustre-OST0001-osc-ffff9d1f98126000@192.168.202.140@tcp:28/4 lens 224/224 e 0 to 0 dl 1767316312 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316302.164562:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f8ad93000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1767316302.164565:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904575488:192.168.202.140@tcp:9:kworker.0 00010000:00010000:2.0:1767316302.774811:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316303.799086:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316304.822839:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316305.849896:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316306.870855:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1767316307.062859:0:2491:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316291/real 1767316291] req@ffff9d1f918d3480 x1853163904574208/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316307.062898:0:2491:0:(client.c:2522:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d1f918d3480 x1853163904574208/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316307 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316307.062907:0:2491:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9d1f980bb800 already not connected (conn 37, was 36): CONNECTING 00000100:00100000:0.0:1767316307.062925:0:2491:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2491:1853163904574208:192.168.202.140@tcp:400:kworker.0 00000100:00080000:1.0:1767316307.255386:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316307.255396:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316307.255398:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316307.255401:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1767316307.894998:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316308.918842:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316309.942942:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316310.966848:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316311.994740:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1767316312.374850:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316312.374878:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316312.374881:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316312.374884:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767316312.375348:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316302/real 1767316302] req@ffff9d1f918d0700 x1853163904575104/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316312.375379:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0700 x1853163904575104/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316312 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316312.375389:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316302/real 1767316302] req@ffff9d1f918d1c00 x1853163904575232/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316312.375401:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1c00 x1853163904575232/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316312 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316312.375413:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316312.375479:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316312.375484:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904575104:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767316312.375499:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767316312.382424:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316312.382434:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767316312.382438:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316312.382448:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1549 00000100:00080000:1.0:1767316312.382460:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1767316312.382467:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767316312.382505:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1880 x1853163904575616/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316312.382523:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316312.382526:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767316312.382528:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316312.382532:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1549 00000100:00080000:1.0:1767316312.382536:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1767316312.382539:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767316312.382549:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767316312.382560:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3800 x1853163904575744/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767316312.383220:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316312.383228:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904575232:192.168.202.140@tcp:38:kworker.0 00000100:00100000:2.0:1767316312.383267:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904575616:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767316312.383374:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904575744:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316313.018699:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1767316313.403570:0:2493:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316297/real 1767316297] req@ffff9d1f967ce300 x1853163904574720/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316313 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316313.403602:0:2493:0:(client.c:2522:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d1f967ce300 x1853163904574720/t0(0) o400->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 224/224 e 0 to 1 dl 1767316313 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316313.403610:0:2493:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9d1f980bb800 already not connected (conn 38, was 36): CONNECTING 00000100:00100000:2.0:1767316313.403627:0:2493:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:45d767c3-7963-411a-be8f-bcbeb1c06075:2493:1853163904574720:192.168.202.140@tcp:400:kworker.0 00000001:02000400:3.0:1767316313.788625:0:47665:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: auster : @@@@@@ FAIL: Restart of mds1 failed! 00010000:00010000:3.0:1767316314.038877:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316315.062791:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767316316.088536:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767316317.116613:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1767316317.495119:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316317.495128:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316317.495131:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316317.495135:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316318.135497:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767316319.159970:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1767316319.223353:0:47994:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9d1f90125348), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1767316319.223367:0:47994:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d1f90125348),name=/ 00000002:00010000:1.0:1767316319.223403:0:47994:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1767316319.223460:0:47994:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccde00/0xa4df836b16491339 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 47994 initiator: MDT0 00010000:00010000:1.0:1767316319.223469:0:47994:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccde00/0xa4df836b16491339 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 47994 initiator: MDT0 00010000:00010000:1.0:1767316319.223479:0:47994:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d1f98126000 lock: ffff9d1f93ccde00/0xa4df836b16491339 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 47994 initiator: MDT0 00000100:00080000:1.0:1767316319.223505:0:47994:0:(client.c:1829:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9d1f918d3100 x1853163904575872/t0(0) o101->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 576/3504 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1767316319.223515:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316320.184714:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316320.246807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316321.208482:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316321.270799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316322.231450:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316322.294994:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316322.618425:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316322.618434:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316322.618437:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316322.618440:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1767316323.261152:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316323.320309:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316324.279207:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316324.343727:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316325.309089:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316325.367302:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316326.327092:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316326.392057:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316327.352282:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316327.415617:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316327.735880:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316327.735891:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316327.735912:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316327.735916:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767316327.737951:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316312/real 1767316312] req@ffff9d1f918d1880 x1853163904575616/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316327.738018:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1880 x1853163904575616/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316327 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316327.738030:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316312/real 1767316312] req@ffff9d1f918d3800 x1853163904575744/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316327.738042:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3800 x1853163904575744/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316327 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316327.738070:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316327.738103:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316327.738108:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904575616:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767316327.738114:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316327.738120:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316327.738123:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904575744:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767316327.738332:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316327.738343:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767316327.738346:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316327.738352:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1560 00000100:00080000:1.0:1767316327.738363:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1767316327.738370:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767316327.738408:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0e00 x1853163904576000/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316327.738437:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316327.738441:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767316327.738443:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316327.738446:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1560 00000100:00080000:1.0:1767316327.738450:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1767316327.738454:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767316327.738463:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767316327.738473:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d2d80 x1853163904576128/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316327.739900:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904576000:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767316327.740038:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904576128:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767316328.377741:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316328.438911:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316329.399834:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316329.462822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316330.423057:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316330.489547:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316331.446961:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316331.510811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316332.470912:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316332.534994:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316332.854934:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316332.854944:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316332.854947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316332.854950:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316333.498708:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316333.558832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316334.519032:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316334.583072:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316335.545726:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316335.606910:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316336.566976:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316336.630953:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316337.590938:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316337.655170:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316337.975069:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316337.975083:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316337.975087:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316337.975090:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316338.614868:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316338.679388:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316339.638848:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316339.702811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316340.663259:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316340.726787:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316341.687126:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316341.750811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316342.714226:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316342.774820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316343.096425:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316343.096435:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316343.096438:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316343.096442:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1767316343.735319:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316343.798904:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316344.758895:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316344.822833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316345.783030:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316345.851038:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316346.807129:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316346.870842:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316347.190818:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316327/real 1767316327] req@ffff9d1f918d0e00 x1853163904576000/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316347.190852:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0e00 x1853163904576000/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316347 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316347.190891:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316327/real 1767316327] req@ffff9d1f918d2d80 x1853163904576128/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316347.190905:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d2d80 x1853163904576128/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316347 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316347.190927:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316347.190959:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316347.190964:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904576000:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767316347.190970:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316347.190976:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316347.190979:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904576128:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767316347.191157:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316347.191164:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767316347.191173:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316347.191179:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1575 00000100:00080000:0.0:1767316347.191191:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1767316347.191199:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767316347.191231:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031f80 x1853163904576256/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767316347.191259:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316347.191262:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767316347.191264:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316347.191268:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1575 00000100:00080000:0.0:1767316347.191272:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1767316347.191276:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767316347.191286:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767316347.191297:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031c00 x1853163904576384/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316347.191534:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904576256:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767316347.191638:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904576384:192.168.202.140@tcp:38:kworker.0 00010000:00010000:2.0:1767316347.831198:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316347.894823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316348.856591:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316348.918848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316349.882492:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316349.943634:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316350.902999:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316350.966833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316351.926862:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316351.990939:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316352.310934:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316352.310959:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316352.310961:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316352.310964:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316352.953128:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316353.015649:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316353.975736:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316354.038825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316354.998891:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316355.062827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316356.022873:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316356.086943:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316357.047308:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316357.110847:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316357.432224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316357.432236:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316357.432240:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316357.432243:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316358.070945:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316358.134756:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316359.099569:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316359.159097:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316360.119990:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316360.182830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316361.147527:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316361.206842:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316362.167001:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316362.230817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316362.550962:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316362.550973:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316362.550976:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316362.550979:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316363.191092:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316363.258525:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316364.214855:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316364.279612:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316365.238920:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316365.303566:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316366.263007:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316366.326801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316367.286957:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316367.350909:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316367.671537:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316367.671548:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316367.671551:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316367.671554:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1767316368.311170:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316368.375436:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316369.335574:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316369.398874:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316370.361874:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316370.422815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316371.382913:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316371.446808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316372.406866:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316372.470805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316372.790794:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316372.790807:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316372.790812:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1767316372.790813:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316347/real 1767316347] req@ffff9d1f91031f80 x1853163904576256/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316372 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767316372.790830:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1767316372.790862:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031f80 x1853163904576256/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316372 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316372.790873:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316347/real 1767316347] req@ffff9d1f91031c00 x1853163904576384/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316372 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316372.790885:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031c00 x1853163904576384/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316372 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767316372.790901:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767316372.790925:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767316372.790929:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904576256:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767316372.790936:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767316372.790943:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767316372.790946:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904576384:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767316372.790971:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316372.790976:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767316372.790979:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316372.790985:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1594 00000100:00080000:3.0:1767316372.790994:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316372.791001:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767316372.791026:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70a80 x1853163904576512/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767316372.791045:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316372.791048:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767316372.791050:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316372.791053:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1594 00000100:00080000:3.0:1767316372.791056:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316372.791059:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767316372.791066:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904576512:192.168.202.140@tcp:250:kworker.0 00000008:00100000:3.0:1767316372.791067:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767316372.791162:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72300 x1853163904576640/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316372.791219:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904576640:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767316373.431511:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316373.494819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316374.456811:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316374.519971:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316375.478942:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316375.542904:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316376.503426:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316376.566772:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316377.530236:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316377.591567:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316377.911560:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316377.911570:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316377.911573:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316377.911597:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316378.551721:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316378.615109:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316379.575050:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316379.638808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316380.598856:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316380.663313:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316381.624609:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316381.686819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316382.647751:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316382.711131:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316383.031047:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316383.031057:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316383.031060:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316383.031063:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316383.672915:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316383.735538:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316384.694829:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316384.759397:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316385.719063:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316385.782828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316386.742893:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316386.808954:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316387.766883:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316387.830909:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316388.150907:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316388.150918:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316388.150921:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316388.150924:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1767316388.790883:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316388.854885:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316389.816099:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316389.878852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316390.841453:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316390.903927:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316391.862981:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316391.932485:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316392.886831:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316392.950809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316393.273499:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316393.273513:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316393.273516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316393.273519:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316393.912846:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316393.974948:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316394.940493:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316394.998979:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316395.959619:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316396.025566:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316396.982889:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316397.046848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316397.366955:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316372/real 1767316372] req@ffff9d1f92f72300 x1853163904576640/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316397.366998:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72300 x1853163904576640/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316397 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316397.367020:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316372/real 1767316372] req@ffff9d1f92f70a80 x1853163904576512/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316397.367031:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70a80 x1853163904576512/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316397 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316397.367050:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316397.367131:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316397.367136:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904576640:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767316397.367141:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767316397.372874:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316397.372886:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767316397.372890:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316397.372900:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1620 00000100:00080000:0.0:1767316397.372941:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316397.372949:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767316397.373007:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030e00 x1853163904576768/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767316397.373027:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316397.373030:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767316397.373032:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316397.373036:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1620 00000100:00080000:0.0:1767316397.373039:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316397.373043:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767316397.373052:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767316397.373064:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032a00 x1853163904576896/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767316397.374636:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316397.374646:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904576512:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767316397.374736:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904576768:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767316397.374852:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904576896:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767316398.006982:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316398.072534:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316399.030892:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316399.097531:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316400.058265:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316400.119269:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316401.079024:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316401.142838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316402.102894:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316402.166836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316402.487650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316402.487661:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316402.487664:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316402.487667:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316403.127888:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316403.191541:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316404.151540:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316404.215121:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316405.174842:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316405.238821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316406.199767:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316406.263533:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316407.223229:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316407.287929:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316407.606814:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316407.606826:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316407.606829:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316407.606841:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316408.259430:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316408.310795:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316409.271249:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316409.335410:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316410.294904:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316410.359385:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316411.319926:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316411.386310:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316412.344216:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316412.407358:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316412.726887:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316412.726900:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316412.726903:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316412.726906:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1767316413.367574:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316413.430802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316414.392043:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316414.456803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316415.416036:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316415.478815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316416.440285:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316416.503709:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767316417.462803:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316417.526843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316417.846932:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316417.846944:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316417.846946:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316417.846950:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316418.488920:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316418.551378:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316419.511374:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316419.574921:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316420.537605:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316420.599190:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316421.566333:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316421.623070:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316422.582915:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316422.646805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316422.966843:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316397/real 1767316397] req@ffff9d1f91030e00 x1853163904576768/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316422 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316422.966883:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030e00 x1853163904576768/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316422 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316422.966893:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316397/real 1767316397] req@ffff9d1f91032a00 x1853163904576896/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316422 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316422.966920:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032a00 x1853163904576896/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316422 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316422.966938:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316422.966945:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316422.966950:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904576768:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767316422.966956:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316422.966960:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316422.966963:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904576896:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767316422.967182:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316422.967187:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767316422.967190:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316422.967195:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1645 00000100:00080000:1.0:1767316422.967204:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316422.967211:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767316422.967242:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1f80 x1853163904577024/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316422.967266:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316422.967269:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767316422.967271:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316422.967275:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1645 00000100:00080000:1.0:1767316422.967278:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316422.967287:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767316422.967297:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767316422.967308:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0000 x1853163904577152/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316422.967767:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904577024:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767316422.967851:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904577152:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316423.607119:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316423.671304:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316424.631789:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316424.694938:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316425.655030:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316425.720365:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316426.679070:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316426.743053:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316427.703129:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316427.767557:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316428.086935:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316428.086945:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316428.086949:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316428.086952:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316428.726948:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316428.790838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316429.751554:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316429.814817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316430.775487:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316430.839846:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316431.799067:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316431.864818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316432.830576:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316432.888634:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316433.210846:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316433.210857:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316433.210860:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316433.210863:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316433.849628:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316433.910929:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316434.871257:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316434.934891:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316435.894990:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316435.958833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316436.919216:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316436.982845:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316437.945990:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316438.006819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316438.326884:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316438.326894:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316438.326897:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316438.326901:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316438.967193:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316439.030820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316439.990934:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316440.055035:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316441.015475:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316441.078821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316442.039992:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316442.102821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316443.062917:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316443.127061:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316443.446773:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316443.446783:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316443.446786:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316443.446789:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316444.087092:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316444.151092:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316445.111288:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316445.175466:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316446.137312:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316446.200584:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316447.159107:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316447.223929:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767316447.542838:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316422/real 1767316422] req@ffff9d1f918d1f80 x1853163904577024/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316447 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316447.542885:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1f80 x1853163904577024/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316447 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316447.542896:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316422/real 1767316422] req@ffff9d1f918d0000 x1853163904577152/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316447 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316447.542919:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0000 x1853163904577152/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316447 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767316447.542940:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767316447.542970:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767316447.542974:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904577024:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767316447.542981:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767316447.542987:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767316447.542990:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904577152:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767316447.543067:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316447.543074:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767316447.543079:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767316447.543085:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1670 00000100:00080000:2.0:1767316447.543093:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767316447.543100:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767316447.543132:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967ce300 x1853163904577280/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767316447.543160:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316447.543164:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767316447.543165:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767316447.543169:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1670 00000100:00080000:2.0:1767316447.543173:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767316447.543176:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767316447.543186:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767316447.543197:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cd500 x1853163904577408/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767316447.543751:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904577280:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767316447.543865:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904577408:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316448.187971:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316448.247151:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316449.207014:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316449.272726:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316450.242531:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316450.294818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316451.254854:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316451.318822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316452.278954:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316452.342783:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316452.663786:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316452.663797:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316452.663810:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316452.663814:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316453.303225:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316453.366892:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316454.327085:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316454.390931:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316455.350971:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316455.414820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316456.374917:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316456.438821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316457.398905:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316457.463081:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316457.784516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316457.784528:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316457.784531:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316457.784534:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316458.422984:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316458.488846:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316459.446936:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316459.510873:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316460.471503:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316460.534850:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316461.494914:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316461.558862:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316462.518931:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316462.584842:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316462.903244:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316462.903255:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316462.903258:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316462.903261:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316463.543325:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316463.606865:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316464.568841:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316464.630855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316465.591807:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316465.655103:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316466.615519:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316466.679808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316467.638898:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316467.705521:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316468.023367:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316468.023377:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316468.023380:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316468.023384:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316468.662967:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316468.727112:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316469.687025:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316469.754099:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316470.711025:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316470.776278:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316471.740600:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316471.799953:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767316472.118818:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316447/real 1767316447] req@ffff9d1f967ce300 x1853163904577280/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316472 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316472.118856:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967ce300 x1853163904577280/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316472 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316472.118868:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316447/real 1767316447] req@ffff9d1f967cd500 x1853163904577408/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316472 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316472.118880:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cd500 x1853163904577408/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316472 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767316472.118899:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767316472.118923:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767316472.118927:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904577280:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767316472.118934:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767316472.118940:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767316472.118944:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904577408:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767316472.119024:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316472.119035:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767316472.119040:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316472.119050:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1695 00000100:00080000:0.0:1767316472.119063:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316472.119072:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767316472.119136:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032a00 x1853163904577536/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767316472.119173:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316472.119178:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767316472.119180:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316472.119184:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1695 00000100:00080000:0.0:1767316472.119188:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316472.119193:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767316472.119204:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767316472.119216:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030e00 x1853163904577664/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767316472.119314:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904577536:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767316472.119445:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904577664:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316472.758926:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316472.822944:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316473.783007:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316473.847166:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316474.807444:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316474.870826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316475.831367:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316475.894983:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316476.855194:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316476.919040:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316477.239108:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316477.239123:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316477.239126:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316477.239155:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316477.879299:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316477.943351:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316478.903276:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316478.966852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316479.927428:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316479.992893:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316480.951627:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316481.014845:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316481.975066:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316482.038847:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316482.364763:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316482.364774:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316482.364777:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316482.364804:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316483.004667:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316483.062880:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316484.023070:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316484.086950:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316485.047559:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316485.110799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316486.072882:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316486.135323:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316487.097128:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316487.158808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316487.479436:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316487.479443:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316487.479446:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316487.479450:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316488.120266:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316488.182894:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316489.142915:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316489.206852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316490.171198:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316490.230800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316491.194290:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316491.254799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316492.220001:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316492.279802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316492.599487:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316492.599500:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316492.599502:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316492.599505:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316493.239764:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316493.302829:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316494.262913:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316494.328662:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316495.288786:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316495.350788:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316496.312392:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316496.374917:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316497.335257:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316497.398969:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767316497.719166:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316472/real 1767316472] req@ffff9d1f91032a00 x1853163904577536/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316497.719201:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032a00 x1853163904577536/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316497 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316497.719212:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316472/real 1767316472] req@ffff9d1f91030e00 x1853163904577664/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767316497.719223:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030e00 x1853163904577664/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316497 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767316497.719244:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767316497.719254:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767316497.719258:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904577536:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767316497.719264:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767316497.719269:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767316497.719272:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904577664:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767316497.721564:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316497.721583:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767316497.721587:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316497.721598:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1719 00000100:00080000:3.0:1767316497.721610:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316497.721618:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767316497.721667:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71c00 x1853163904577792/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767316497.721744:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316497.721748:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767316497.721749:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316497.721753:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1719 00000100:00080000:3.0:1767316497.721772:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316497.721776:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767316497.721785:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767316497.721796:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73100 x1853163904577920/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767316497.721947:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904577792:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767316497.722042:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904577920:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767316498.358943:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316498.425075:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316499.386746:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316499.446894:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316500.407383:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316500.472231:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316501.431385:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316501.495707:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316502.454909:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316502.521068:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316502.838925:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316502.838936:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316502.838938:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316502.838941:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316503.478904:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316503.543398:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316504.503432:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316504.566826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316505.527388:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316505.591026:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316506.556114:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316506.617825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316507.575675:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316507.638811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316507.958803:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316507.958813:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316507.958815:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316507.958819:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316508.599709:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316508.662792:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316509.622985:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316509.686792:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316510.648328:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316510.710942:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316511.671783:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316511.734836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316512.694930:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316512.758883:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316513.078842:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316513.078852:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316513.078855:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316513.078859:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316513.719090:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316513.782782:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316514.743406:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316514.806826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316515.767931:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316515.830897:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316516.799837:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316516.854848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316517.814932:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316517.878802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316518.200290:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316518.200300:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316518.200303:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316518.200307:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316518.839185:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316518.904364:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316519.862906:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316519.926814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316520.890330:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316520.950978:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316521.910919:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316521.975497:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316522.295119:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316497/real 1767316497] req@ffff9d1f92f71c00 x1853163904577792/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316522.295203:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71c00 x1853163904577792/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316522 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316522.295214:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316497/real 1767316497] req@ffff9d1f92f73100 x1853163904577920/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316522.295225:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73100 x1853163904577920/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316522 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316522.295243:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316522.295270:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316522.295274:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904577792:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767316522.295279:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316522.295285:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316522.295288:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904577920:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767316522.295326:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316522.295332:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767316522.295336:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316522.295342:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1745 00000100:00080000:0.0:1767316522.295351:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316522.295357:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767316522.295385:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031c00 x1853163904578048/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316522.295452:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904578048:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767316522.295582:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316522.295586:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767316522.295589:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316522.295601:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1745 00000100:00080000:0.0:1767316522.295605:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316522.295609:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767316522.295617:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767316522.295631:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031f80 x1853163904578176/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316522.297007:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904578176:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767316522.935220:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316522.999083:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316523.962112:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316524.022994:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316524.983517:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316525.047600:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316526.008203:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316526.070902:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316527.030916:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316527.094808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316527.416311:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316527.416322:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316527.416325:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316527.416347:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316528.056282:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316528.118833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316529.079272:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316529.142845:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316530.104266:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316530.166811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316531.127912:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316531.191107:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316532.150961:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316532.214809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316532.534861:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316532.534872:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316532.534876:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316532.534879:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316533.175422:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316533.239671:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316534.199043:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316534.264185:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316535.223712:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316535.288360:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316536.247537:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316536.311288:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316537.273425:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316537.334804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316537.656045:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316537.656054:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316537.656057:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316537.656061:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316538.294925:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316538.358797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316539.319292:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316539.383103:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316540.342895:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316540.407251:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316541.368162:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316541.430808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316542.391969:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316542.455050:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316542.775258:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316542.775269:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316542.775271:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316542.775274:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316543.415374:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316543.479624:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316544.439115:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316544.502932:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316545.462949:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316545.528283:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316546.486925:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316546.550844:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316547.513835:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316547.574846:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316547.894815:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316522/real 1767316522] req@ffff9d1f91031f80 x1853163904578176/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316547 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316547.894869:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031f80 x1853163904578176/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316547 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316547.894881:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316522/real 1767316522] req@ffff9d1f91031c00 x1853163904578048/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316547 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316547.894891:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031c00 x1853163904578048/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316547 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316547.894909:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316547.894917:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316547.894921:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904578176:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767316547.894925:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316547.894929:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316547.894932:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904578048:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767316547.896323:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316547.896332:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767316547.896336:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316547.896343:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1769 00000100:00080000:3.0:1767316547.896353:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316547.896360:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767316547.896402:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73100 x1853163904578304/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316547.896483:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904578304:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767316547.897090:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316547.897096:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767316547.897098:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316547.897103:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1769 00000100:00080000:3.0:1767316547.897108:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316547.897113:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767316547.897124:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767316547.897152:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71c00 x1853163904578432/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316547.901330:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904578432:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316548.534912:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316548.598812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316549.558913:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316549.623511:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316550.582910:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316550.646820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316551.608120:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316551.670820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316552.631462:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316552.694827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316553.015475:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316553.015497:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316553.015500:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316553.015504:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316553.655111:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316553.719581:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316554.678913:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316554.742813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316555.703779:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316555.774415:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316556.734715:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316556.790984:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316557.751292:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316557.814917:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316558.134932:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316558.134944:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316558.134947:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316558.134951:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316558.777710:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316558.839210:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316559.804352:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316559.863028:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316560.824229:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316560.886900:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316561.847814:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316561.913960:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316562.872625:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316562.935004:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316563.255085:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316563.255101:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316563.255105:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316563.255108:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316563.895335:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316563.959256:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316564.918979:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316564.983106:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316565.944801:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316566.006824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316566.967079:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316567.035175:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316567.993384:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316568.054868:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316568.374933:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316568.374943:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316568.374947:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316568.375140:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316569.017531:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316569.080479:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316570.039337:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316570.102877:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316571.063225:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316571.126816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316572.087010:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316572.150940:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316572.470887:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316547/real 1767316547] req@ffff9d1f92f71c00 x1853163904578432/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316572 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316572.470924:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71c00 x1853163904578432/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316572 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316572.470936:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316547/real 1767316547] req@ffff9d1f92f73100 x1853163904578304/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316572 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316572.470948:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73100 x1853163904578304/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316572 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316572.470967:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316572.470993:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316572.470998:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904578432:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767316572.471003:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316572.471021:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316572.471025:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904578304:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767316572.471391:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316572.471397:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767316572.471400:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316572.471406:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1795 00000100:00080000:1.0:1767316572.471415:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316572.471421:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767316572.471466:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0000 x1853163904578560/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316572.471645:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316572.471649:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767316572.471651:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316572.471656:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1795 00000100:00080000:1.0:1767316572.471660:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316572.471664:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767316572.471715:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767316572.471739:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1f80 x1853163904578688/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316572.471870:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904578560:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767316572.471968:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904578688:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316573.111016:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316573.174848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316574.134991:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316574.198810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316575.158922:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316575.222826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316576.182902:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316576.246900:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316577.208124:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316577.270814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316577.591118:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316577.591146:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316577.591150:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316577.591153:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316578.230905:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316578.294997:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316579.255054:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316579.321269:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316580.278906:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316580.342825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316581.303401:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316581.366830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316582.326952:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316582.391236:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316582.712277:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316582.712288:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316582.712291:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316582.712312:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316583.351221:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316583.414926:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316584.374951:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316584.438865:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316585.398948:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316585.462872:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316586.422971:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316586.488404:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316587.447845:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316587.511161:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316587.830996:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316587.831006:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316587.831009:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316587.831012:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316588.470910:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316588.534814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316589.495234:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316589.558980:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316590.518931:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316590.582818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316591.543707:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316591.609096:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316592.569201:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316592.630823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316592.950854:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316592.950864:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316592.950868:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316592.950871:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316593.591018:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316593.654815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316594.615050:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316594.680935:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316595.639466:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316595.704178:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316596.663148:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316596.726797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316597.046833:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316572/real 1767316572] req@ffff9d1f918d0000 x1853163904578560/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316597.046885:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0000 x1853163904578560/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316597 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316597.046897:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316572/real 1767316572] req@ffff9d1f918d1f80 x1853163904578688/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316597.046931:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1f80 x1853163904578688/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316597 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316597.046949:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316597.046979:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316597.046983:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904578560:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767316597.046989:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316597.046995:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316597.046998:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904578688:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767316597.047096:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316597.047102:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767316597.047105:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316597.047110:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1820 00000100:00080000:3.0:1767316597.047118:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316597.047125:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767316597.047154:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70a80 x1853163904578816/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767316597.047179:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316597.047188:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767316597.047190:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316597.047194:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1820 00000100:00080000:3.0:1767316597.047199:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316597.047202:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767316597.047212:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767316597.047224:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72300 x1853163904578944/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316597.048060:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904578816:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767316597.048168:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904578944:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316597.687301:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316597.750837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316598.710992:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316598.774935:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316599.735176:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316599.799109:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316600.768917:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316600.824539:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316601.788515:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316601.846884:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316602.166935:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316602.166947:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316602.166950:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316602.166954:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316602.806963:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316602.870947:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316603.831970:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316603.894779:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316604.856392:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316604.918835:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316605.879060:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316605.942805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316606.904306:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316606.966904:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316607.291835:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316607.291847:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316607.291850:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316607.291853:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316607.927355:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316607.992958:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316608.950994:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316609.014834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316609.975447:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316610.043302:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316610.999745:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316611.062821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316612.023115:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316612.086827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316612.406878:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316612.406907:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316612.406910:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316612.406913:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316613.050942:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316613.111069:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316614.070915:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316614.135536:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316615.094915:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316615.158834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316616.118867:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316616.183262:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316617.142898:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316617.206823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316617.526782:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316617.526794:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316617.526797:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316617.526799:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316618.166904:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316618.230805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316619.190983:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316619.254848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316620.214891:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316620.278805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316621.247509:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316621.305981:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316622.262924:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316622.329646:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316622.646864:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316597/real 1767316597] req@ffff9d1f92f70a80 x1853163904578816/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316622.646896:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70a80 x1853163904578816/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316622.646907:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316597/real 1767316597] req@ffff9d1f92f72300 x1853163904578944/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316622.646919:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72300 x1853163904578944/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316622.646936:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316622.646955:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316622.646959:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904578816:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767316622.646965:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316622.646971:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316622.646974:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904578944:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767316622.647083:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316622.647088:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767316622.647091:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316622.647096:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1844 00000100:00080000:1.0:1767316622.647105:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316622.647111:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767316622.647144:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1f80 x1853163904579072/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316622.647188:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316622.647191:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767316622.647193:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316622.647197:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1844 00000100:00080000:1.0:1767316622.647201:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316622.647205:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767316622.647213:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767316622.647225:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0000 x1853163904579200/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316622.647478:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904579072:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767316622.647580:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904579200:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767316623.287182:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316623.350791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316624.311903:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316624.374803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316625.336644:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316625.399044:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316626.360518:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316626.422809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316627.382938:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316627.446813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316627.767924:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316627.767938:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316627.767979:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316627.767982:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316628.408841:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316628.471071:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316629.436391:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316629.494823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316630.457996:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316630.518975:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316631.483330:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316631.542826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316632.505358:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316632.567058:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316632.886920:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316632.886931:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316632.886933:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316632.886937:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316633.530748:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316633.590965:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316634.563528:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316634.614797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316635.575031:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316635.638829:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316636.598974:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316636.663292:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316637.623001:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316637.687489:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316638.006802:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316638.006812:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316638.006815:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316638.006818:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316638.650712:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316638.710802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316639.672230:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316639.734808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316640.695096:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316640.758797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316641.718909:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316641.782789:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316642.746451:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316642.807660:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316643.127489:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316643.127507:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316643.127510:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316643.127514:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316643.766864:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316643.830833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316644.790973:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316644.854797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316645.815009:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316645.879483:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316646.841645:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316646.902810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316647.222875:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316622/real 1767316622] req@ffff9d1f918d1f80 x1853163904579072/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316647 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316647.222913:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1f80 x1853163904579072/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316647 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316647.222924:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316622/real 1767316622] req@ffff9d1f918d0000 x1853163904579200/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316647 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316647.222936:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0000 x1853163904579200/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316647 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316647.222978:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316647.223011:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316647.223016:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904579072:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767316647.223022:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316647.223027:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316647.223030:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904579200:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767316647.223485:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316647.223493:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767316647.223498:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316647.223506:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1870 00000100:00080000:1.0:1767316647.223517:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316647.223533:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767316647.223887:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d2d80 x1853163904579328/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316647.223918:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316647.223922:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767316647.223924:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316647.223929:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1870 00000100:00080000:1.0:1767316647.223934:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316647.223938:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767316647.223947:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767316647.223961:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0e00 x1853163904579456/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316647.224044:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904579328:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767316647.224162:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904579456:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767316647.864291:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316647.926926:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316648.888235:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316648.952410:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316649.915279:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316649.975264:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316650.935546:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316650.998886:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316651.959169:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316652.022815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316652.343240:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316652.343249:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316652.343252:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316652.343256:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316652.982945:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316653.046803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316654.007292:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316654.071364:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316655.031016:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316655.094814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316656.055418:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316656.118902:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316657.079202:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316657.142809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316657.463347:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316657.463359:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316657.463362:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316657.463366:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316658.103070:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316658.166849:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316659.127417:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316659.193182:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316660.150928:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316660.214805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316661.176645:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316661.238815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316662.200403:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316662.262798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316662.582851:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316662.582864:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316662.582867:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316662.582871:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316663.223446:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316663.287297:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316664.247227:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316664.310822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316665.270950:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316665.338128:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316666.294940:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316666.358801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316667.325365:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316667.382859:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316667.703184:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316667.703195:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316667.703198:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316667.703219:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316668.343525:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316668.406791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316669.368219:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316669.430909:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316670.393005:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316670.454765:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316671.414922:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316671.482000:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316672.438880:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316672.502812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316672.823290:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316672.823307:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316672.823311:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316672.823314:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767316672.823464:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316647/real 1767316647] req@ffff9d1f918d2d80 x1853163904579328/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316672 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316672.823497:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d2d80 x1853163904579328/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316672 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316672.823507:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316647/real 1767316647] req@ffff9d1f918d0e00 x1853163904579456/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316672 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316672.823518:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0e00 x1853163904579456/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316672 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316672.823564:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316672.823586:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316672.823590:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904579328:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767316672.823597:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316672.823602:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316672.823605:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904579456:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767316672.823752:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316672.823757:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767316672.823760:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316672.823766:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1894 00000100:00080000:1.0:1767316672.823774:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316672.823780:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767316672.823808:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3800 x1853163904579584/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316672.823840:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316672.823844:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767316672.823846:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316672.823849:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1894 00000100:00080000:1.0:1767316672.823852:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316672.823856:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767316672.823866:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767316672.823877:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1880 x1853163904579712/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316672.823945:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904579584:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767316672.824056:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904579712:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316673.469248:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316673.526804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316674.487015:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316674.550822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316675.511600:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316675.574838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316676.536883:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316676.598872:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316677.558979:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316677.623199:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316677.943015:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316677.943025:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316677.943028:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316677.943031:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316678.582945:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316678.647100:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316679.612039:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316679.670815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316680.630981:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316680.694827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316681.654899:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316681.718980:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316682.679148:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316682.742949:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316683.062837:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316683.062868:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316683.062871:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316683.062875:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316683.702956:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316683.770752:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316684.727237:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316684.791091:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316685.751021:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316685.815118:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316686.777914:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316686.838817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316687.799042:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316687.862966:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316688.186753:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316688.186765:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316688.186769:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316688.186772:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316688.823312:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316688.886767:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316689.849660:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316689.912452:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316690.871112:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316690.934821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316691.895302:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316691.959750:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316692.920424:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316692.982804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316693.303544:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316693.303554:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316693.303556:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316693.303559:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316693.942898:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316694.006813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316694.966931:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316695.030887:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316695.991023:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316696.054824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316697.018221:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316697.078957:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767316697.398803:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316672/real 1767316672] req@ffff9d1f918d3800 x1853163904579584/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316697.398840:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3800 x1853163904579584/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316697 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316697.398850:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316672/real 1767316672] req@ffff9d1f918d1880 x1853163904579712/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316697.398861:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1880 x1853163904579712/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316697 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767316697.398878:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767316697.398905:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767316697.398909:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904579584:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767316697.398916:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767316697.398922:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767316697.398925:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904579712:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767316697.399024:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316697.399029:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767316697.399033:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316697.399038:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1920 00000100:00080000:1.0:1767316697.399047:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316697.399053:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767316697.399088:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1c00 x1853163904579840/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316697.399112:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316697.399129:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767316697.399131:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316697.399135:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1920 00000100:00080000:1.0:1767316697.399140:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316697.399143:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767316697.399153:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767316697.399165:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0700 x1853163904579968/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767316697.399575:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904579840:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767316697.399724:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904579968:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316698.039972:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316698.103281:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316699.062934:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316699.127928:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316700.088293:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316700.152154:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316701.111260:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316701.174873:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316702.137662:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316702.200199:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316702.518877:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316702.518890:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316702.518893:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316702.518897:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316703.160320:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316703.223761:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316704.185563:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316704.247217:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316705.208284:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316705.274365:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316706.233545:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316706.294816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316707.256174:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316707.322230:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316707.638995:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316707.639006:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316707.639008:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316707.639011:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316708.285406:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316708.342819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316709.302915:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316709.371883:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316710.327230:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316710.390852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316711.351513:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316711.414826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316712.375250:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316712.438813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316712.758902:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316712.758913:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316712.758916:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316712.758919:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316713.400173:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316713.464108:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316714.422852:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316714.486821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316715.447416:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316715.511031:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316716.470876:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316716.535053:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316717.494909:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316717.558803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316717.879763:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316717.879776:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316717.879778:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316717.879782:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316718.518887:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316718.582815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316719.543019:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316719.606986:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316720.566902:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316720.631591:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316721.591243:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316721.655191:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316722.615187:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316722.680062:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316722.998893:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316697/real 1767316697] req@ffff9d1f918d1c00 x1853163904579840/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316722.998974:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316722.998980:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316722.998982:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316722.998985:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767316722.999503:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1c00 x1853163904579840/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316722 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316722.999515:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316697/real 1767316697] req@ffff9d1f918d0700 x1853163904579968/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316722.999530:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0700 x1853163904579968/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316722 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316722.999546:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316722.999745:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316722.999750:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904579840:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767316722.999757:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316722.999763:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316722.999766:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904579968:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767316722.999867:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316722.999871:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767316722.999874:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316722.999879:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1945 00000100:00080000:0.0:1767316722.999889:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316722.999895:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767316722.999939:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033800 x1853163904580096/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767316722.999967:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316722.999970:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767316722.999972:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316722.999975:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1945 00000100:00080000:0.0:1767316722.999979:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316722.999983:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767316722.999992:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767316723.000004:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033b80 x1853163904580224/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316723.000428:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904580096:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767316723.000769:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904580224:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767316723.639954:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316723.705155:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316724.662920:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316724.727148:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316725.686915:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316725.750810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316726.714103:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316726.774795:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316727.734903:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316727.798809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316728.119046:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316728.119056:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316728.119059:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316728.119062:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316728.775809:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316728.822817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316729.784499:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316729.848470:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316730.807723:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316730.872290:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316731.832623:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316731.894815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316732.855447:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316732.919037:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316733.238910:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316733.238922:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316733.238925:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316733.238928:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316733.878921:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316733.942810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316734.909563:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316734.967144:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316735.927006:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316735.991986:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316736.951288:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316737.016880:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316737.974936:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316738.039075:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316738.358828:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316738.358841:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316738.358844:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316738.358847:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316738.999526:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316739.064021:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316740.022942:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316740.086824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316741.050930:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316741.110805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316742.070899:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316742.134814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316743.100348:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316743.158801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316743.480237:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316743.480248:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316743.480251:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316743.480255:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316744.119492:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316744.183525:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316745.143027:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316745.206842:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316746.167248:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316746.230807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316747.191949:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316747.254811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767316747.510793:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316722/real 1767316723] req@ffff9d1f91033800 x1853163904580096/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316747.510830:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033800 x1853163904580096/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316747 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767316747.510853:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767316747.510882:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767316747.510887:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904580096:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767316747.510988:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316747.510994:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767316747.510996:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316747.511001:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1970 00000100:00080000:1.0:1767316747.511011:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316747.511017:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767316747.511054:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3480 x1853163904580352/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316747.511078:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316747.511082:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1767316747.511423:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904580352:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767316748.221966:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316748.278818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767316748.534873:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316723/real 1767316723] req@ffff9d1f91033b80 x1853163904580224/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316748.534908:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033b80 x1853163904580224/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316748 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767316748.534930:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767316748.534955:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767316748.534958:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904580224:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767316748.535262:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316748.535286:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316748.535289:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316748.535293:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767316748.535296:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316748.535301:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1970 00000100:00080000:1.0:1767316748.535312:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316748.535318:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767316748.535331:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767316748.535362:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1180 x1853163904580480/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767316748.535545:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904580480:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316749.243163:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316749.302836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316750.262906:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316750.326831:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316751.287585:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316751.350828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316752.312451:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316752.375313:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316752.566785:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316752.566797:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316752.566799:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316752.566802:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316753.334943:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316753.398771:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316753.591427:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316753.591439:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316753.591441:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316753.591444:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316754.359357:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316754.424610:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316755.384224:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316755.448637:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316756.406909:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316756.470808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316757.431091:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316757.495424:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316758.455950:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316758.518771:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316758.845323:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316758.845336:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316758.845339:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316758.845343:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316759.480418:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316759.542750:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316760.502915:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316760.566798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316761.527712:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316761.590814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316762.558246:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316762.616056:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316763.575415:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316763.638817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316763.959409:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316763.959426:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316763.959430:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316763.959435:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316764.599266:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316764.662855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316765.623151:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316765.686992:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316766.649252:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316766.712841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316767.670869:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316767.734872:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316768.697027:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316768.761281:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316769.079170:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316769.079181:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316769.079183:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316769.079186:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316769.720561:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316769.782824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316770.742879:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316770.806829:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316771.767894:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316771.830814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767316772.022808:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316747/real 1767316747] req@ffff9d1f918d3480 x1853163904580352/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316772 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316772.022845:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3480 x1853163904580352/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316772 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767316772.022884:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767316772.022913:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767316772.022918:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904580352:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767316772.023023:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316772.023028:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767316772.023031:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316772.023036:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 1995 00000100:00080000:1.0:1767316772.023045:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316772.023051:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767316772.023085:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1500 x1853163904580608/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767316772.023157:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904580608:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767316772.023279:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316772.023284:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316772.790968:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316772.854794:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767316773.048123:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316748/real 1767316748] req@ffff9d1f918d1180 x1853163904580480/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316773 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316773.048154:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1180 x1853163904580480/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316773 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767316773.048174:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767316773.048204:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767316773.048210:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904580480:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767316773.048287:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316773.048303:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316773.048306:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316773.048310:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767316773.048313:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316773.048319:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 1996 00000100:00080000:1.0:1767316773.048329:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316773.048335:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767316773.048346:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767316773.048371:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d2300 x1853163904580736/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767316773.048753:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904580736:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316773.821329:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316773.880250:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316774.847119:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316774.902786:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316775.864146:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316775.926859:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316776.887250:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316776.954568:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316777.078961:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316777.078976:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316777.078980:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316777.078983:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316777.910936:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316777.974865:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316778.104535:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316778.104551:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316778.104555:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316778.104558:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316778.937840:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316778.998830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316779.959557:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316780.022813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316780.983120:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316781.046873:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316782.006929:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316782.070803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316783.031034:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316783.094767:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316783.414878:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316783.414909:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316783.414912:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316783.414916:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316784.060258:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316784.118803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316785.078925:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316785.143189:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316786.103560:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316786.166811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316787.127060:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316787.190892:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316788.151255:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316788.214915:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316788.535564:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316788.535571:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316788.535573:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316788.535576:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316789.174913:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316789.238812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316790.202502:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316790.264887:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316791.223195:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316791.286801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316792.246915:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316792.314009:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316793.274805:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316793.335383:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316793.654858:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316793.654867:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316793.654870:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316793.654887:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316794.294918:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316794.359801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316795.319071:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316795.382860:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316796.343547:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316796.408120:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316797.371199:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316797.430918:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767316797.558865:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316772/real 1767316772] req@ffff9d1f918d1500 x1853163904580608/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316797.558893:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1500 x1853163904580608/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316797 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767316797.558909:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767316797.558929:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767316797.558933:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904580608:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767316797.558973:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316797.558978:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767316797.558981:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316797.558986:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2019 00000100:00080000:0.0:1767316797.559014:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316797.559163:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767316797.559210:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033480 x1853163904580864/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767316797.559235:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316797.559239:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1767316797.559411:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904580864:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767316798.390960:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316798.454864:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767316798.583235:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316773/real 1767316773] req@ffff9d1f918d2300 x1853163904580736/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767316798.583274:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d2300 x1853163904580736/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316798 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767316798.583295:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767316798.583344:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767316798.583349:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904580736:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767316798.583744:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316798.583748:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316798.583750:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316798.583753:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767316798.583755:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316798.583759:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2020 00000100:00080000:0.0:1767316798.583768:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316798.583773:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767316798.583782:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767316798.583814:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030700 x1853163904580992/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767316798.584257:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904580992:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767316799.415359:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316799.479122:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316800.438940:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316800.503374:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316801.462824:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316801.527021:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316802.487150:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316802.552478:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316802.615178:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316802.615190:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316802.615193:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316802.615196:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316803.511359:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316803.575003:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316803.638800:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316803.638811:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316803.638814:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316803.638818:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316804.535665:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316804.598843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316805.559860:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316805.622852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316806.583411:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316806.647802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316807.606918:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316807.670885:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316808.631277:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316808.694838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316809.015324:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316809.015334:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316809.015336:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316809.015339:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316809.654999:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316809.721198:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316810.679053:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316810.742913:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316811.704833:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316811.767931:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316812.726903:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316812.790817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316813.750901:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316813.814852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316814.137130:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316814.137169:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316814.137173:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316814.137176:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316814.774904:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316814.839100:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316815.798995:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316815.862838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316816.822904:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316816.886800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316817.853345:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316817.910909:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316818.871028:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316818.934817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316819.255618:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316819.255629:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316819.255632:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316819.255636:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316819.894927:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316819.959286:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316820.918910:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316820.982889:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316821.942916:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316822.009315:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767316822.071237:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316797/real 1767316797] req@ffff9d1f91033480 x1853163904580864/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316822 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316822.071273:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033480 x1853163904580864/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316822 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316822.071294:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767316822.071322:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767316822.071326:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904580864:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767316822.071414:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316822.071419:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767316822.071422:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767316822.071443:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2045 00000100:00080000:2.0:1767316822.071452:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767316822.071458:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767316822.071497:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cea00 x1853163904581120/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767316822.071519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316822.071529:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767316822.071703:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904581120:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767316822.967078:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316823.030921:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767316823.094823:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316798/real 1767316798] req@ffff9d1f91030700 x1853163904580992/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316823.094851:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030700 x1853163904580992/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316823 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316823.094868:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767316823.094942:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767316823.094946:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904580992:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767316823.095088:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316823.095110:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316823.095112:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316823.095115:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767316823.095118:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316823.095122:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2046 00000100:00080000:3.0:1767316823.095130:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316823.095150:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767316823.095161:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767316823.095188:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70e00 x1853163904581248/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767316823.095279:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904581248:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767316823.990919:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316824.054777:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316825.014957:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316825.078870:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316826.040759:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316826.104881:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316827.067570:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316827.126804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316827.126826:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316827.126832:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316827.126834:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316827.126837:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316828.096107:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1767316828.151486:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316828.151497:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316828.151501:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316828.151504:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767316828.152113:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316829.111366:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316829.174768:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316830.136877:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316830.198984:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316831.158942:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316831.222990:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316832.182973:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316832.247803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316833.206908:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316833.270818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316833.591715:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316833.591726:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316833.591730:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316833.591737:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316834.234057:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316834.296838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316835.254973:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316835.321237:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316836.278925:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316836.342868:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316837.303971:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316837.367018:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316838.327244:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316838.390818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316838.711745:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316838.711757:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316838.711760:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316838.711764:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316839.350866:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316839.414809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316840.375234:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316840.438819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316841.402637:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316841.462811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316842.423702:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316842.486809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316843.447270:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316843.510823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316843.832472:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316843.832484:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316843.832487:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316843.832491:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316844.472998:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316844.535206:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316845.497456:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316845.558949:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316846.520474:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316846.582875:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316847.549457:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316847.606819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316847.607258:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316822/real 1767316822] req@ffff9d1f967cea00 x1853163904581120/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316847 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316847.607321:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cea00 x1853163904581120/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316847 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316847.607339:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316847.607364:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316847.607369:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904581120:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767316847.607543:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316847.607550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767316847.607553:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316847.607558:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2069 00000100:00080000:0.0:1767316847.607567:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316847.607573:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767316847.607605:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030700 x1853163904581376/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767316847.607640:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316847.607644:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767316847.609321:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904581376:192.168.202.140@tcp:250:kworker.0 00010000:00010000:1.0:1767316848.566868:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316848.630801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316848.630810:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316823/real 1767316823] req@ffff9d1f92f70e00 x1853163904581248/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316848 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316848.630845:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70e00 x1853163904581248/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316848 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316848.630865:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316848.630893:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316848.630899:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904581248:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767316848.630983:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316848.631005:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316848.631008:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316848.631012:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767316848.631014:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316848.631020:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2070 00000100:00080000:3.0:1767316848.631029:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316848.631035:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767316848.631048:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767316848.631075:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70380 x1853163904581504/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316848.631142:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904581504:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316849.590907:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316849.655207:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316850.614911:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316850.678810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316851.639185:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316851.703164:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316852.662785:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316852.662796:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316852.662799:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316852.662802:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316852.665868:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316852.727452:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316853.687033:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316853.687044:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316853.687047:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316853.687050:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316853.690231:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316853.751109:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316854.712406:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316854.777216:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316855.735245:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316855.798817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316856.761497:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316856.822801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316857.783557:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316857.846801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316858.810699:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316858.874258:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316859.192345:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316859.192355:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316859.192357:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316859.192361:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316859.831458:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316859.895726:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316860.855321:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316860.918816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316861.878957:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316861.942821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316862.902940:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316862.967600:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316863.926952:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316863.992486:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316864.310810:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316864.310820:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316864.310823:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316864.310825:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316864.951331:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316865.014845:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316865.974993:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316866.039089:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316867.003606:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316867.062944:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316868.023419:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316868.087707:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316869.047636:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316869.112773:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316869.433013:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316869.433023:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316869.433027:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316869.433030:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316870.070979:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316870.135237:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316871.095554:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316871.158778:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767316872.119143:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316847/real 1767316847] req@ffff9d1f91030700 x1853163904581376/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316872 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316872.119177:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030700 x1853163904581376/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316872 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316872.119196:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767316872.119240:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767316872.119245:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904581376:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767316872.119317:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316872.119322:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767316872.119325:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316872.119331:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2095 00000100:00080000:1.0:1767316872.119340:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316872.119346:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767316872.119382:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0380 x1853163904581632/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316872.119409:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316872.119413:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767316872.119464:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904581632:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767316872.123530:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316872.182836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767316873.142839:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316848/real 1767316848] req@ffff9d1f92f70380 x1853163904581504/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316873 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316873.142885:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70380 x1853163904581504/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316873 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316873.142905:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767316873.142931:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767316873.142936:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904581504:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767316873.143109:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316873.143115:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316873.143117:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316873.143120:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767316873.143123:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316873.143129:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2096 00000100:00080000:1.0:1767316873.143138:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316873.143144:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767316873.143155:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767316873.143195:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3b80 x1853163904581760/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1767316873.143356:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1767316873.143394:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904581760:192.168.202.140@tcp:38:kworker.0 00000100:00100000:2.0:1767316873.207569:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316874.167583:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316874.230816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316875.191028:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316875.254829:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316876.214927:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316876.278833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316877.174843:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316877.174863:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316877.174866:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316877.174869:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316877.238927:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316877.303269:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316878.199163:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316878.199174:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316878.199177:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316878.199180:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316878.286848:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316878.326962:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316879.288885:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316879.351286:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316880.311269:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316880.374856:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316881.334845:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316881.401738:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316882.360975:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316882.425415:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316883.255187:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316883.255198:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316883.255216:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316883.255219:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316883.383011:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316883.448072:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316884.410375:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316884.470772:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316885.432260:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316885.496561:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316886.454956:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316886.518810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316887.479471:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316887.545465:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316888.374807:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316888.374816:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316888.374819:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316888.374822:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316888.502903:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316888.567949:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316889.529561:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316889.594266:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316890.551542:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316890.617031:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316891.574875:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316891.638905:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316892.599512:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316892.662853:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316893.496310:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316893.496325:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316893.496328:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316893.496332:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316893.623647:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316893.690131:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316894.648671:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316894.711440:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316895.671089:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316895.735243:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316896.695639:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316896.758816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316897.655148:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316872/real 1767316872] req@ffff9d1f918d0380 x1853163904581632/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316897.655210:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0380 x1853163904581632/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316897 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316897.655234:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316897.655263:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316897.655267:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904581632:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767316897.655461:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316897.655467:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767316897.655470:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316897.655476:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2119 00000100:00080000:0.0:1767316897.655485:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316897.655491:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767316897.655525:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030700 x1853163904581888/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767316897.655547:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316897.655550:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767316897.655899:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904581888:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767316897.719093:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316897.782953:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316898.678839:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316873/real 1767316873] req@ffff9d1f918d3b80 x1853163904581760/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316898 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316898.678880:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3b80 x1853163904581760/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316898 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316898.678904:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316898.678933:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316898.678937:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904581760:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767316898.679032:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316898.679038:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316898.679041:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316898.679044:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767316898.679047:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316898.679053:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2120 00000100:00080000:0.0:1767316898.679063:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316898.679069:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767316898.679081:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767316898.679110:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033480 x1853163904582016/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316898.679191:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904582016:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316898.742909:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316898.806824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316899.767760:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316899.830812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316900.792253:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316900.854818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316901.815028:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316901.883426:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316902.710956:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316902.710972:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316902.710976:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316902.710980:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316902.838987:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316902.903575:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316903.739555:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316903.739571:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316903.739574:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316903.739578:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316903.862960:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316903.929483:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316904.890143:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316904.950823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316905.910932:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316905.974832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316906.937641:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316906.998854:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316907.960512:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316908.023581:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767316908.858169:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316908.858181:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767316908.858184:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316908.858188:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316908.989037:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316909.047490:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316910.007108:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316910.070825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316911.030917:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316911.096387:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316912.055293:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316912.121240:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316913.084194:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316913.142814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316913.974972:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316913.974984:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316913.974987:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316913.974991:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316914.103340:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316914.166782:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316915.127307:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316915.191414:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316916.151990:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316916.217674:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316917.180708:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316917.239226:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316918.199452:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316918.262826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316919.095216:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316919.095226:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316919.095230:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316919.095233:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316919.225554:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316919.287669:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316920.246908:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316920.312729:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316921.271276:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316921.336817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316922.166956:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316897/real 1767316897] req@ffff9d1f91030700 x1853163904581888/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316922.166989:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030700 x1853163904581888/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316922 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316922.167009:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316922.167033:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316922.167037:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904581888:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767316922.167089:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316922.167096:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767316922.167099:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316922.167105:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2145 00000100:00080000:3.0:1767316922.167114:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316922.167121:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767316922.167157:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70700 x1853163904582144/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316922.167256:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904582144:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767316922.168363:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316922.168370:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316922.296621:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316922.361288:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316923.190930:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316898/real 1767316898] req@ffff9d1f91033480 x1853163904582016/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316923.191002:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033480 x1853163904582016/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316923 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316923.191024:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316923.191095:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316923.191101:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904582016:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767316923.193818:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316923.193827:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316923.193830:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316923.193834:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767316923.193856:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767316923.193868:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2146 00000100:00080000:0.0:1767316923.193879:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767316923.193887:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767316923.193900:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767316923.193934:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033100 x1853163904582272/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316923.194015:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904582272:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316923.321774:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316923.384597:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316924.342904:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316924.406786:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316925.366959:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316925.430850:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316926.392268:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316926.456395:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316927.222908:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316927.222918:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316927.222921:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316927.222924:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316927.415291:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316927.478841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316928.248660:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316928.248671:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316928.248673:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316928.248716:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316928.439390:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316928.502861:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316929.463216:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316929.526792:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316930.488390:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316930.552208:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316931.510886:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316931.575111:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316932.537421:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316932.599769:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316933.430794:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316933.430804:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316933.430807:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316933.430811:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316933.566638:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316933.622811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316934.583070:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316934.647431:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316935.607026:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316935.670846:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316936.630911:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316936.694836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316937.655037:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316937.720483:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316938.550807:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316938.550818:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316938.550821:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316938.550825:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316938.679279:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316938.743902:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316939.703331:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316939.766799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316940.726880:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316940.790793:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316941.750904:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316941.815634:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316942.775013:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316942.839560:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316943.670904:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316943.670926:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316943.670929:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316943.670932:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316943.798890:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316943.863513:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316944.824331:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316944.888328:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316945.846885:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316945.910827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316946.871002:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316946.934816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316947.703438:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316922/real 1767316922] req@ffff9d1f92f70700 x1853163904582144/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316947 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316947.703489:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70700 x1853163904582144/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316947 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316947.703510:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316947.703542:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316947.703546:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904582144:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767316947.703650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316947.703657:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767316947.703660:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316947.703666:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2169 00000100:00080000:3.0:1767316947.703676:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316947.703702:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767316947.703736:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73800 x1853163904582400/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767316947.703763:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316947.703767:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767316947.703883:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904582400:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767316947.894874:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316947.960173:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316948.729428:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316923/real 1767316923] req@ffff9d1f91033100 x1853163904582272/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316948.729465:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033100 x1853163904582272/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316948 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316948.729488:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316948.729521:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316948.729526:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904582272:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767316948.729596:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316948.729602:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316948.729605:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316948.729608:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767316948.729611:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316948.729617:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2170 00000100:00080000:3.0:1767316948.729638:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316948.729645:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767316948.729655:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767316948.729706:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71180 x1853163904582528/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316948.729876:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904582528:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767316948.924061:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316948.982819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316949.942989:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316950.006891:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316950.966992:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767316951.031242:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316951.991633:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316952.056241:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316952.759241:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316952.759253:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316952.759256:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316952.759259:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316953.015270:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316953.078800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316953.785115:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316953.785125:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316953.785128:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316953.785131:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316954.039006:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316954.102791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316955.064912:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316955.126809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316956.087274:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316956.151076:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316957.110969:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316957.175205:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316958.134929:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316958.200713:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316959.031737:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316959.031748:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316959.031751:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316959.031754:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316959.158882:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316959.222827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316960.183737:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316960.246810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316961.206970:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316961.270880:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316962.230891:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316962.294835:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316963.254897:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316963.318890:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316964.150901:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316964.150913:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316964.150916:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316964.150919:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316964.280229:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316964.342820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316965.303897:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316965.366996:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316966.326970:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316966.390808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316967.350996:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316967.415247:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316968.377694:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316968.438820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316969.272246:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316969.272256:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316969.272258:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316969.272279:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316969.398947:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316969.463843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316970.423885:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316970.486812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316971.446972:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316971.510830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316972.214984:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316947/real 1767316947] req@ffff9d1f92f73800 x1853163904582400/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316972 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316972.215015:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73800 x1853163904582400/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316972 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316972.215034:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316972.215053:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316972.215057:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904582400:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767316972.215493:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316972.215499:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767316972.215502:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316972.215507:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2195 00000100:00080000:3.0:1767316972.215517:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316972.215522:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767316972.215552:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73480 x1853163904582656/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767316972.215574:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316972.215577:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767316972.215705:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904582656:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767316972.470891:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316972.535565:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767316973.238811:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316948/real 1767316948] req@ffff9d1f92f71180 x1853163904582528/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767316973.238845:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71180 x1853163904582528/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316973 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767316973.238864:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767316973.238895:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767316973.238900:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904582528:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767316973.238974:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316973.238979:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316973.238982:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316973.238985:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767316973.238988:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316973.238993:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2196 00000100:00080000:3.0:1767316973.239002:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316973.239008:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767316973.239018:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767316973.239048:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71f80 x1853163904582784/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767316973.239153:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904582784:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767316973.494928:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316973.558913:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316974.519474:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316974.593431:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316975.546801:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316975.606815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316976.567807:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316976.630896:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316977.285234:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316977.285248:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316977.285251:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316977.285254:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316977.591026:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316977.657296:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767316978.294833:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316978.294863:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767316978.294867:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767316978.294869:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316978.615447:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316978.679320:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316979.638918:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316979.702809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316980.664571:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316980.728336:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316981.687187:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316981.750963:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316982.711553:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316982.774764:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316983.607305:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316983.607315:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316983.607318:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316983.607321:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767316983.735192:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316983.799050:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316984.758926:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316984.822836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316985.783285:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316985.848088:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316986.808709:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316986.871155:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316987.831265:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316987.896458:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767316988.727448:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316988.727460:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316988.727463:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316988.727465:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767316988.855087:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316988.919073:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767316989.878912:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316989.943585:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316990.903060:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316990.968586:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316991.927373:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316991.990839:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316992.951036:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316993.014814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767316993.846861:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316993.846888:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767316993.846892:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767316993.846896:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767316993.975096:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316994.038838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316994.999229:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316995.064036:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767316996.023174:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316996.091818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767316997.049948:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316997.110823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767316997.750833:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316972/real 1767316972] req@ffff9d1f92f73480 x1853163904582656/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316997 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316997.750864:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73480 x1853163904582656/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767316997 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316997.750884:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767316997.750903:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767316997.750907:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904582656:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767316997.751090:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316997.751095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767316997.751098:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767316997.751103:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2219 00000100:00080000:1.0:1767316997.751112:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767316997.751120:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767316997.751151:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3b80 x1853163904582912/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767316997.751175:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767316997.751179:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767316997.751319:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904582912:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767316998.076662:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316998.134813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767316998.775927:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316973/real 1767316973] req@ffff9d1f92f71f80 x1853163904582784/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767316998.775997:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71f80 x1853163904582784/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767316998 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767316998.776019:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767316998.776098:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767316998.776103:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904582784:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767316998.785675:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316998.785697:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767316998.785700:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767316998.785704:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767316998.785708:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767316998.785719:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2220 00000100:00080000:3.0:1767316998.785729:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767316998.785754:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767316998.785768:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767316998.785800:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72680 x1853163904583040/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767316998.786411:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904583040:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767316999.098297:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767316999.160451:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317000.119594:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317000.183511:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317001.143981:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317001.206813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317002.174940:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317002.233834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317002.807349:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317002.807361:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317002.807364:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317002.807367:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317003.190915:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317003.254830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317003.830910:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317003.830920:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317003.830923:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317003.830926:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317004.214894:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317004.278825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317005.239455:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317005.302808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317006.265642:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317006.326839:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317007.287547:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317007.350967:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317008.310957:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317008.375998:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317009.206951:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317009.206962:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317009.206964:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317009.206968:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317009.334945:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317009.401881:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317010.358942:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317010.422819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317011.383307:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317011.446827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317012.407414:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317012.470814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317013.430879:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317013.495054:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317014.327787:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317014.327798:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317014.327819:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317014.327822:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317014.455394:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317014.518821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317015.479387:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317015.543112:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317016.503214:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317016.569509:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317017.527265:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317017.590855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317018.552067:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317018.614832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317019.447579:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317019.447591:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317019.447594:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317019.447598:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317019.577562:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317019.638891:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317020.600545:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317020.663123:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317021.624825:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317021.687495:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317022.262849:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316997/real 1767316997] req@ffff9d1f918d3b80 x1853163904582912/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317022 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317022.262889:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3b80 x1853163904582912/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317022 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317022.262918:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317022.262947:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317022.262951:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904582912:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317022.263092:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317022.263097:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317022.263101:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317022.263106:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2245 00000100:00080000:1.0:1767317022.263117:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317022.263122:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317022.263151:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0380 x1853163904583168/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767317022.263176:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317022.263186:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317022.263248:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904583168:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317022.647262:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317022.712868:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317023.287896:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316998/real 1767316998] req@ffff9d1f92f72680 x1853163904583040/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317023.287930:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72680 x1853163904583040/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317023 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317023.287951:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317023.287980:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317023.287984:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904583040:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317023.288072:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317023.288077:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317023.288080:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317023.288083:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317023.288086:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317023.288091:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2246 00000100:00080000:1.0:1767317023.288100:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317023.288106:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317023.288117:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317023.288149:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d2300 x1853163904583296/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317023.288338:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904583296:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317023.673707:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317023.736212:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317024.697273:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317024.758793:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317025.719710:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317025.782778:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317026.742841:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317026.810178:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317027.318858:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317027.318869:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317027.318873:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317027.318876:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317027.767935:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317027.830836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317028.342784:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317028.342792:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317028.342794:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317028.342811:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317028.794509:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317028.854895:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317029.817508:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317029.880841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317030.839107:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317030.903040:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317031.863078:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317031.926829:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317032.886898:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317032.950818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317033.782826:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317033.782835:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317033.782838:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317033.782841:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317033.912612:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317033.975067:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317034.934972:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317034.999449:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317035.960095:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317036.027171:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317036.982926:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317037.046805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317038.008507:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317038.070955:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317038.902796:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317038.902806:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317038.902809:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317038.902812:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317039.032390:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317039.095814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317040.054902:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317040.119070:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317041.078881:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317041.148633:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317042.105228:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317042.167748:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317043.126915:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317043.191424:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317044.022978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317044.022989:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317044.022992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317044.022995:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317044.151803:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317044.214962:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317045.174918:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317045.238891:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317046.202626:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317046.262801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317047.225470:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317047.287143:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317047.799894:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317022/real 1767317022] req@ffff9d1f918d0380 x1853163904583168/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317047.799931:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0380 x1853163904583168/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317047 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317047.799963:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317047.799992:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317047.799997:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904583168:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317047.800160:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317047.800166:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317047.800169:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317047.800175:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2269 00000100:00080000:1.0:1767317047.800184:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317047.800191:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317047.800224:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1500 x1853163904583424/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767317047.800249:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317047.800254:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317047.800424:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904583424:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317048.248252:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317048.312014:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317048.824302:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317023/real 1767317023] req@ffff9d1f918d2300 x1853163904583296/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317048 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317048.824342:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d2300 x1853163904583296/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317048 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317048.824364:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317048.824395:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317048.824400:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904583296:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317048.824750:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317048.824756:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317048.824758:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317048.824761:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317048.824765:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317048.824771:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2270 00000100:00080000:1.0:1767317048.824780:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317048.824786:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317048.824797:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317048.824829:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1180 x1853163904583552/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767317048.824912:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904583552:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767317049.271633:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317049.334895:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317050.294931:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317050.362718:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317051.319031:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317051.382821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317052.342960:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317052.406813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317052.855517:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317052.855545:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317052.855547:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317052.855549:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317053.368719:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317053.430830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317053.879407:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317053.879419:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317053.879422:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317053.879425:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317054.392467:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317054.454801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317055.419186:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317055.478805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317056.440144:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317056.507427:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317057.463339:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317057.526786:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317058.487817:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317058.550912:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317059.383508:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317059.383519:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317059.383522:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317059.383525:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317059.523538:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317059.575125:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317060.538739:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317060.598832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317061.558967:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317061.624464:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317062.583926:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317062.646806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317063.607884:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317063.670860:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317064.502897:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317064.502909:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317064.502912:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317064.502915:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317064.631026:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317064.694816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317065.657708:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317065.718804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317066.680412:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317066.742804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317067.703097:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317067.766814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317068.726926:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317068.790860:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317069.625479:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317069.625491:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317069.625495:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317069.625498:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317069.751385:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317069.814789:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317070.775399:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317070.838799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317071.800712:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317071.862820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317072.310820:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317047/real 1767317047] req@ffff9d1f918d1500 x1853163904583424/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317072.310855:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1500 x1853163904583424/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317072 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317072.310892:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317072.310918:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317072.310924:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904583424:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317072.311164:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317072.311169:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317072.311182:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317072.311194:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2295 00000100:00080000:1.0:1767317072.311204:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317072.311211:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317072.311723:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3480 x1853163904583680/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767317072.311750:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317072.311754:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317072.311883:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904583680:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317072.822916:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317072.887191:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317073.334857:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317048/real 1767317048] req@ffff9d1f918d1180 x1853163904583552/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317073 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317073.334893:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1180 x1853163904583552/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317073 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317073.334915:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317073.334939:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317073.334944:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904583552:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767317073.334983:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317073.334990:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317073.334993:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317073.334996:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767317073.335000:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317073.335025:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2296 00000100:00080000:2.0:1767317073.335036:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317073.335042:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767317073.335053:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767317073.335078:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cea00 x1853163904583808/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317073.335145:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904583808:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767317073.848411:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317073.910820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317074.870987:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317074.936333:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317075.894862:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317075.958804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317076.919977:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317076.982824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317077.366996:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317077.367006:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317077.367009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317077.367012:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317077.942923:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317078.006864:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317078.974419:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317079.030897:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317079.991925:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317080.054801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317081.015255:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317081.078834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317082.038907:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317082.102811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317082.424167:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317082.424197:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317082.424200:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317082.424203:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317083.062909:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317083.127045:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317084.088529:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317084.151504:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317085.111154:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317085.174758:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317086.134924:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317086.198831:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317087.161051:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317087.223529:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317087.543148:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317087.543162:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317087.543165:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317087.543169:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317088.183142:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317088.246797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317089.207209:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317089.270827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317090.231023:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317090.294997:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317091.254925:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317091.318816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317092.278887:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317092.343821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317092.665273:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317092.665284:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317092.665287:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317092.665290:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317093.302907:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317093.366796:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317094.332033:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317094.390817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317095.351260:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317095.414812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317096.376731:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317096.439320:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317097.399279:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317097.463582:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317097.782930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317097.782941:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317097.782944:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317097.782947:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317097.848557:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317072/real 1767317072] req@ffff9d1f918d3480 x1853163904583680/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317097 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317097.848592:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3480 x1853163904583680/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317097 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317097.848611:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317097.848639:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317097.848651:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904583680:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317097.849508:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317097.849514:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317097.849518:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317097.849524:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2319 00000100:00080000:1.0:1767317097.849535:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317097.849541:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317097.849576:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0700 x1853163904583936/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767317097.849603:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317097.849608:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317097.850066:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904583936:192.168.202.140@tcp:250:kworker.0 00010000:00010000:1.0:1767317098.430367:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317098.486800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317098.873476:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317073/real 1767317073] req@ffff9d1f967cea00 x1853163904583808/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317098.873507:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cea00 x1853163904583808/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317098 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317098.873523:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317098.873546:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317098.873549:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904583808:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317098.873626:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317098.873630:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317098.873632:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317098.873657:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317098.873660:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317098.873664:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2320 00000100:00080000:1.0:1767317098.873672:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317098.873693:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317098.873702:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317098.873724:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1c00 x1853163904584064/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767317098.873808:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904584064:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317099.449193:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317099.510812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317100.472013:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317100.535051:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317101.496136:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317101.559082:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317102.525508:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317102.582804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317102.902751:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317102.902759:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317102.902761:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317102.902764:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317103.543199:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317103.606806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317103.928365:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317103.928375:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317103.928378:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317103.928381:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317104.568015:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317104.631038:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317105.590956:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317105.655939:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317106.614971:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317106.678855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317107.639080:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317107.702999:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317108.663066:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317108.726840:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317109.046847:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317109.046858:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317109.046879:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317109.046882:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317109.687032:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317109.754046:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317110.711109:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317110.775260:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317111.739095:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317111.798796:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317112.758922:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317112.823399:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317113.782965:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317113.849255:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317114.167361:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317114.167372:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317114.167375:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317114.167378:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317114.807938:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317114.870830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317115.830913:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317115.896249:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317116.855440:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317116.923553:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317117.878976:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317117.942798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317118.903733:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317118.966840:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317119.286850:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317119.286866:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317119.286871:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317119.286876:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317119.928881:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317119.990807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317120.951156:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317121.015175:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317121.975987:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317122.044924:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317122.359462:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317097/real 1767317097] req@ffff9d1f918d0700 x1853163904583936/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317122.359511:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0700 x1853163904583936/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317122 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317122.359540:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317122.359565:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317122.359570:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904583936:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767317122.360653:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317122.360663:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767317122.360668:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317122.360855:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2345 00000100:00080000:3.0:1767317122.360869:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317122.360877:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767317122.360937:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72d80 x1853163904584192/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767317122.361032:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317122.361036:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317122.361139:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904584192:192.168.202.140@tcp:250:kworker.0 00010000:00010000:1.0:1767317122.999067:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317123.063602:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317123.382900:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317098/real 1767317098] req@ffff9d1f918d1c00 x1853163904584064/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317123 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317123.382948:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1c00 x1853163904584064/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317123 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317123.382976:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317123.383016:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317123.383022:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904584064:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767317123.383116:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317123.383124:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317123.383127:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317123.383132:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767317123.383221:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317123.383228:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2346 00000100:00080000:3.0:1767317123.383242:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317123.383251:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767317123.383265:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767317123.383310:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72a00 x1853163904584320/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767317123.383386:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904584320:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317124.023520:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317124.088318:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317125.048788:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317125.111930:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317126.072065:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317126.134880:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317127.096288:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317127.158837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317127.416774:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317127.416785:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317127.416788:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317127.416791:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317128.119413:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317128.183060:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317128.438851:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317128.438859:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317128.438861:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317128.438864:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317129.143425:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317129.206813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317130.168871:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317130.230800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317131.191860:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317131.254833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317132.215016:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317132.279596:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317133.238987:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317133.302915:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317133.624912:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317133.624923:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317133.624926:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317133.624930:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317134.272914:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317134.326894:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317135.286904:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317135.350812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317136.310991:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317136.374831:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317137.335587:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317137.398858:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317138.359065:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317138.422965:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317138.742837:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317138.742848:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317138.742851:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317138.742854:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317139.382892:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317139.446820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317140.407527:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317140.470853:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317141.431703:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317141.494822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317142.455573:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317142.519444:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317143.483953:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317143.542963:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317143.862953:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317143.862963:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317143.862966:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317143.862969:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317144.503019:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317144.566843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317145.526925:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317145.591216:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317146.554153:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317146.615044:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317147.578625:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317147.639726:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317147.894824:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317122/real 1767317122] req@ffff9d1f92f72d80 x1853163904584192/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317147 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317147.894864:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72d80 x1853163904584192/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317147 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317147.894887:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317147.894918:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317147.894924:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904584192:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767317147.895030:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317147.895035:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767317147.895039:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317147.895045:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2370 00000100:00080000:2.0:1767317147.895055:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317147.895063:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767317147.895101:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cea00 x1853163904584448/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767317147.895128:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317147.895132:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317147.895204:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904584448:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317148.598901:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317148.662761:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317148.918774:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317123/real 1767317123] req@ffff9d1f92f72a00 x1853163904584320/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317148 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317148.918811:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72a00 x1853163904584320/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317148 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317148.918833:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317148.918888:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317148.918893:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904584320:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767317148.919113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317148.919119:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317148.919122:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317148.919132:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767317148.919135:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317148.919140:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2371 00000100:00080000:2.0:1767317148.919150:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317148.919156:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767317148.919168:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767317148.919200:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cc700 x1853163904584576/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317148.919357:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904584576:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317149.624098:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317149.687437:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317150.664353:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317150.720389:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317151.674486:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317151.736035:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317152.694978:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317152.759441:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317152.951001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317152.951012:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317152.951015:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317152.951018:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317153.718956:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317153.782856:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317153.974860:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317153.974871:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317153.974874:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317153.974877:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317154.746276:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317154.806970:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317155.766926:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317155.830799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317156.791361:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317156.854895:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317157.817225:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317157.880590:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317158.839550:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317158.902820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317159.225708:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317159.225720:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317159.225723:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317159.225726:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317159.862925:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317159.926830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317160.886974:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317160.950819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317161.910925:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317161.974860:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317162.934880:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317162.998812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317163.959133:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317164.023028:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317164.344231:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317164.344242:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317164.344246:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317164.344250:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317164.988841:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317165.046987:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317166.007185:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317166.071600:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317167.030994:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317167.096953:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317168.054896:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317168.118817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317169.086165:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317169.142825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317169.463001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317169.463014:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317169.463017:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317169.463021:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317170.102956:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317170.166841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317171.132191:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317171.192009:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317172.151032:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317172.214833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317172.406830:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317147/real 1767317147] req@ffff9d1f967cea00 x1853163904584448/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317172.406872:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cea00 x1853163904584448/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317172 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317172.406892:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317172.406920:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317172.406924:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904584448:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767317172.407065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317172.407076:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767317172.407079:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317172.407085:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2395 00000100:00080000:2.0:1767317172.407094:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317172.407100:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767317172.407129:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cce00 x1853163904584704/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767317172.407155:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317172.407159:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317172.407215:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904584704:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317173.176077:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317173.238829:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317173.430847:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317148/real 1767317148] req@ffff9d1f967cc700 x1853163904584576/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317173 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317173.430882:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cc700 x1853163904584576/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317173 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317173.430904:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317173.430931:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317173.430936:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904584576:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767317173.431028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317173.431035:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317173.431037:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317173.431040:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767317173.431043:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317173.431048:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2396 00000100:00080000:2.0:1767317173.431057:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317173.431063:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767317173.431074:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767317173.431105:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cc000 x1853163904584832/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317173.431488:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904584832:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317174.198924:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317174.264747:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317175.223013:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317175.286771:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317176.246938:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317176.311504:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317177.270889:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317177.334779:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317177.463174:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317177.463183:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317177.463186:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317177.463190:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317178.298101:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317178.358756:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317178.486812:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317178.486824:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317178.486844:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317178.486848:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317179.319113:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317179.382819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317180.343008:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317180.406817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317181.366867:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317181.430919:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317182.390852:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317182.455114:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317183.415000:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317183.478817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317183.800138:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317183.800149:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317183.800152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317183.800155:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317184.438975:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317184.503274:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317185.463045:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317185.527223:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317186.490835:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317186.550808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317187.510939:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317187.575206:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317188.535108:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317188.598821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317188.919715:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317188.919726:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317188.919730:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317188.919733:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317189.559754:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317189.622809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317190.585661:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317190.646903:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317191.606948:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317191.671277:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317192.631894:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317192.694789:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317193.657949:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317193.718816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317194.039954:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317194.039965:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317194.039968:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317194.039988:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317194.680273:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317194.743585:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317195.703259:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317195.766816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317196.726982:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317196.790903:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317197.750975:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317197.814878:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317197.942972:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317172/real 1767317172] req@ffff9d1f967cce00 x1853163904584704/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317197.943010:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cce00 x1853163904584704/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317197 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317197.943053:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317197.943117:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317197.943122:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904584704:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317197.944873:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317197.944884:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317197.944887:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317197.944898:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2420 00000100:00080000:1.0:1767317197.944908:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317197.944914:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317197.944950:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1880 x1853163904584960/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767317197.944982:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317197.944986:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317197.945070:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904584960:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317198.775050:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317198.838927:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317198.967036:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317173/real 1767317173] req@ffff9d1f967cc000 x1853163904584832/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317198.967072:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cc000 x1853163904584832/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317198 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317198.967162:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317198.967231:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317198.967237:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904584832:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317198.984590:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317198.984601:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317198.984604:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317198.984608:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317198.984612:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317198.984623:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2421 00000100:00080000:1.0:1767317198.984654:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317198.984662:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317198.984673:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317198.984721:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3800 x1853163904585088/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317198.985615:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904585088:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317199.800187:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317199.862822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317200.823799:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317200.886818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317201.847817:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317201.912440:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317202.872975:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317202.935087:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317202.998809:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317202.998820:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317202.998824:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317202.998828:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317203.894898:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317203.958812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317204.027769:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317204.027779:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317204.027782:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317204.027786:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317204.920259:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317204.984411:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317205.943099:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317206.008959:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317206.967597:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317207.030810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317207.994256:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317208.054772:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317209.014945:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317209.079586:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317209.398950:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317209.398960:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317209.398963:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317209.398966:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317210.039134:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317210.102810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317211.063059:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317211.126802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317212.086909:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317212.150925:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317213.110980:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317213.174812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317214.138865:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317214.199735:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317214.520505:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317214.520516:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317214.520519:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317214.520522:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317215.159007:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317215.222800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317216.183241:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317216.246830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317217.207127:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317217.270791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317218.231129:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317218.294892:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317219.258758:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317219.318863:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317219.640027:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317219.640038:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317219.640041:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317219.640044:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317220.279667:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317220.343820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317221.302866:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317221.366969:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317222.327273:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317222.390855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317222.455929:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317197/real 1767317197] req@ffff9d1f918d1880 x1853163904584960/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317222.455977:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1880 x1853163904584960/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317222 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317222.455999:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317222.456097:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317222.456103:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904584960:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317222.459219:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317222.459230:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317222.459234:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317222.459244:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2445 00000100:00080000:1.0:1767317222.459255:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317222.459280:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317222.459463:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0e00 x1853163904585216/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317222.459549:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904585216:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317222.459672:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317222.460215:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317223.350865:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317223.415094:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317223.480278:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317198/real 1767317198] req@ffff9d1f918d3800 x1853163904585088/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317223 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317223.480317:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3800 x1853163904585088/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317223 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317223.481230:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317223.481328:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317223.481333:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904585088:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317223.483714:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317223.483726:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317223.483729:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317223.483733:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317223.483738:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317223.483750:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2446 00000100:00080000:1.0:1767317223.483765:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317223.486270:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317223.486291:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317223.486338:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d2d80 x1853163904585344/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317223.487422:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904585344:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317224.375705:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317224.440446:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317225.398804:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317225.463027:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317226.424620:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317226.486754:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317227.450392:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317227.510894:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317227.510918:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317227.511232:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317227.511254:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317227.511258:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317228.471420:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1767317228.534889:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317228.534900:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317228.534904:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317228.534906:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317228.535116:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317229.494903:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317229.558970:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317230.520044:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317230.583243:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317231.544763:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317231.609639:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317232.567222:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317232.630893:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317233.591744:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317233.654834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317233.979250:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317233.979264:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317233.979268:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317233.979271:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317234.623727:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317234.678873:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317235.639207:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317235.702874:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317236.662949:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317236.727116:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317237.686935:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317237.753319:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317238.710937:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317238.774999:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317239.095747:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317239.095757:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317239.095760:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317239.095782:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317239.735600:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317239.801855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317240.761840:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317240.823326:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317241.784786:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317241.846897:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317242.806938:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317242.871732:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317243.831193:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317243.895062:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317244.214813:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317244.214823:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317244.214826:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317244.214830:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317244.854899:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317244.918805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317245.879192:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317245.946168:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317246.903068:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317246.968219:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317247.927240:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317247.990811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317247.990996:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317222/real 1767317222] req@ffff9d1f918d0e00 x1853163904585216/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317247 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317247.991034:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0e00 x1853163904585216/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317247 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317247.991053:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317247.991076:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317247.991081:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904585216:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317247.991391:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317247.991396:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317247.991399:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317247.991404:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2470 00000100:00080000:1.0:1767317247.991413:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317247.991420:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317247.991452:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0000 x1853163904585472/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767317247.991476:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317247.991480:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317247.991537:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904585472:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317248.950975:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1767317249.014958:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317223/real 1767317223] req@ffff9d1f918d2d80 x1853163904585344/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317249.015004:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d2d80 x1853163904585344/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317248 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317249.015023:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317249.015053:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317249.015057:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904585344:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317249.015368:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317249.015373:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317249.015376:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317249.015379:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317249.015382:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317249.015388:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2471 00000100:00080000:1.0:1767317249.015396:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317249.015402:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317249.015420:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317249.015445:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1f80 x1853163904585600/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317249.015525:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904585600:192.168.202.140@tcp:38:kworker.0 00000100:00100000:2.0:1767317249.016375:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317249.975005:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317250.039188:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317251.001365:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317251.062819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317252.023531:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317252.089375:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317253.046926:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317253.046937:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317253.046939:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317253.046942:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317253.047330:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317253.110896:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317254.074306:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1767317254.074508:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317254.074515:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317254.074518:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317254.074521:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317254.134848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317255.095209:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317255.158841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317256.121009:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317256.185307:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317257.142937:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317257.206809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317258.169061:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317258.230978:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317259.190969:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317259.254819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317259.574793:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317259.574804:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317259.574807:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317259.574810:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317260.216248:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317260.279509:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317261.239048:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317261.302815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317262.264372:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317262.327550:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317263.287170:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317263.350868:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317264.311149:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317264.375937:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317264.696180:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317264.696190:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317264.696193:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317264.696196:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317265.334944:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317265.398872:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317266.359102:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317266.422866:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317267.382921:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317267.446793:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317268.408711:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317268.473783:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317269.431024:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317269.494760:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317269.814853:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317269.814865:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317269.814868:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317269.814871:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317270.455446:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317270.523558:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317271.478930:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317271.542810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317272.502970:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317247/real 1767317247] req@ffff9d1f918d0000 x1853163904585472/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317272 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317272.503011:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0000 x1853163904585472/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317272 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317272.503062:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317272.503156:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317272.503161:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904585472:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317272.503597:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1767317272.504303:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317272.504373:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317272.504377:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317272.504385:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2495 00000100:00080000:1.0:1767317272.504396:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317272.504402:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317272.504446:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0000 x1853163904585728/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767317272.504992:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904585728:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317272.507487:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317272.507498:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317272.567389:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317273.527717:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317273.590816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317274.551527:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317274.551996:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317249/real 1767317249] req@ffff9d1f918d1f80 x1853163904585600/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317274 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317274.552028:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1f80 x1853163904585600/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317274 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317274.552049:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317274.552076:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317274.552080:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904585600:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317274.552136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317274.552142:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317274.552157:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317274.552161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317274.552164:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317274.552170:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2496 00000100:00080000:1.0:1767317274.552179:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317274.552186:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317274.552196:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317274.552226:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d2d80 x1853163904585856/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767317274.553536:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904585856:192.168.202.140@tcp:38:kworker.0 00000100:00100000:2.0:1767317274.615227:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317275.576968:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317275.638801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317276.598866:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317276.662805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317277.559469:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317277.559481:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317277.559485:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317277.559488:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317277.623183:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317277.688344:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317278.648744:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317278.711633:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317279.607699:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317279.607710:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317279.607713:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317279.607716:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317279.672810:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317279.734810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317280.695201:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317280.758797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317281.718977:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317281.784634:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317282.742909:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317282.806804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317283.766959:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317283.833932:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317284.662903:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317284.663282:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317284.663286:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317284.663289:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317284.792897:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317284.856917:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317285.819050:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317285.879014:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317286.838937:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317286.902825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317287.862901:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317287.929957:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317288.887393:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317288.950827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317289.782800:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317289.782810:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317289.782813:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317289.782816:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317289.911660:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317289.974986:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317290.935048:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317291.000976:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317291.959086:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317292.022939:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317292.983237:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317293.046918:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317294.007741:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317294.070811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317294.903814:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317294.903826:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317294.903828:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317294.903831:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317295.034990:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317295.094838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317296.055722:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317296.118778:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317297.014893:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317272/real 1767317272] req@ffff9d1f918d0000 x1853163904585728/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317297 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317297.014931:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0000 x1853163904585728/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317297 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317297.014953:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317297.014983:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317297.014987:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904585728:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767317297.015236:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317297.015242:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767317297.015245:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317297.015250:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2520 00000100:00080000:3.0:1767317297.015260:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317297.015298:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767317297.015330:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71500 x1853163904585984/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767317297.015352:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317297.015357:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317297.015420:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904585984:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317297.082389:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317297.142878:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317298.103068:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317298.166802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317299.062772:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317274/real 1767317274] req@ffff9d1f918d2d80 x1853163904585856/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317299 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317299.062810:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d2d80 x1853163904585856/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317299 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317299.062831:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317299.062916:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317299.062930:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904585856:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317299.063118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317299.063124:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317299.063127:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317299.063130:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317299.063133:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317299.063138:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2522 00000100:00080000:1.0:1767317299.063148:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317299.063155:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317299.063166:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317299.063211:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0e00 x1853163904586112/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767317299.063298:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904586112:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317299.132264:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317299.191964:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317300.152183:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317300.214888:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317301.175595:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317301.238797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317302.073576:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317302.073587:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317302.073589:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317302.073593:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317302.199202:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317302.262865:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317303.225641:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317303.286809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317304.118893:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317304.118901:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317304.118904:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317304.118906:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317304.247723:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317304.310817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317305.271570:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317305.334827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317306.297331:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317306.358784:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317307.320710:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317307.386077:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317308.343986:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317308.406818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317309.238964:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317309.238974:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317309.238991:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317309.238994:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317309.366920:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317309.432188:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317310.393056:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317310.458179:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317311.415469:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317311.479624:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317312.439612:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317312.502823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317313.464838:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317313.526825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317314.358770:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317314.358781:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317314.358801:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317314.358804:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317314.488496:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317314.551641:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317315.510993:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317315.579200:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317316.539708:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317316.599364:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317317.558917:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317317.623477:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317318.582940:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317318.646822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317319.479013:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317319.479024:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317319.479027:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317319.479030:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317319.607021:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317319.670898:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317320.634442:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317320.694972:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317321.654931:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317321.719597:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317322.550873:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317297/real 1767317297] req@ffff9d1f92f71500 x1853163904585984/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317322.550907:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71500 x1853163904585984/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317322 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317322.550926:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317322.550946:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317322.550950:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904585984:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767317322.551092:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317322.551097:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767317322.551100:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767317322.551122:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2544 00000100:00080000:0.0:1767317322.551131:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767317322.551137:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767317322.551167:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030380 x1853163904586240/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767317322.551198:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317322.551203:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1767317322.551472:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904586240:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317322.679065:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317322.744099:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317323.703200:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317323.768039:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767317324.599022:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317299/real 1767317299] req@ffff9d1f918d0e00 x1853163904586112/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317324 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767317324.599057:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0e00 x1853163904586112/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317324 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767317324.599078:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767317324.599169:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767317324.599174:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904586112:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767317324.600013:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317324.600019:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317324.600022:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317324.600026:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767317324.600030:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767317324.600036:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2546 00000100:00080000:0.0:1767317324.600046:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767317324.600054:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767317324.600066:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767317324.600099:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031880 x1853163904586368/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767317324.601564:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904586368:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317324.726903:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317324.792237:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317325.751014:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317325.814825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317326.778451:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317326.840060:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317327.606881:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317327.606891:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317327.606894:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317327.606897:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317327.798980:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317327.862859:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317328.822922:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317328.886825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317329.657136:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317329.657147:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317329.657150:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317329.657153:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317329.851856:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317329.910840:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317330.870983:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317330.934864:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317331.894848:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317331.958813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317332.920578:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317332.982820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317333.943122:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317334.006810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317334.839002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317334.839012:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317334.839024:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317334.839028:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317334.967202:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317335.032024:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317335.990928:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317336.055100:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317337.015236:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317337.078815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317338.039863:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317338.104506:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317339.062896:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317339.127872:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317339.959057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317339.959069:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317339.959072:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317339.959075:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317340.086936:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317340.150948:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317341.110996:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317341.174803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317342.135003:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317342.199265:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317343.160007:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317343.222813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317344.183580:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317344.246903:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317345.079054:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317345.079066:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317345.079070:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317345.079073:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317345.207463:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317345.270821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317346.231741:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317346.294816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317347.068407:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317322/real 1767317322] req@ffff9d1f91030380 x1853163904586240/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317347.068443:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030380 x1853163904586240/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317347 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317347.068464:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317347.068498:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317347.068502:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904586240:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767317347.076292:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317347.076304:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767317347.076308:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767317347.076319:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2570 00000100:00080000:0.0:1767317347.076329:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767317347.076336:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767317347.076374:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032d80 x1853163904586496/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767317347.076409:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317347.076413:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317347.076527:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904586496:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317347.256433:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317347.318836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317348.279367:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317348.344256:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317349.110986:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317324/real 1767317324] req@ffff9d1f91031880 x1853163904586368/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317349 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317349.111023:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031880 x1853163904586368/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317349 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317349.111044:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317349.111073:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317349.111077:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904586368:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317349.111242:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317349.111248:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317349.111251:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317349.111254:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317349.111257:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317349.111263:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2572 00000100:00080000:1.0:1767317349.111272:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317349.111294:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317349.111305:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317349.111550:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3800 x1853163904586624/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767317349.111635:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904586624:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317349.304127:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317349.366818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317350.326955:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317350.390815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317351.350901:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317351.416422:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317352.124766:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317352.124777:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317352.124780:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317352.124783:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317352.374900:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317352.438854:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317353.398962:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317353.463062:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317354.167135:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317354.167146:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317354.167150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317354.167153:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317354.422894:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317354.488133:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317355.446975:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317355.510851:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317356.471444:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317356.537734:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317357.496887:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317357.558891:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317358.519962:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317358.582879:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317359.414845:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317359.414855:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317359.414868:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317359.414872:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317359.544978:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317359.606844:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317360.566944:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317360.630850:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317361.593197:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317361.654824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317362.614932:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317362.678742:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317363.639153:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317363.703269:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317364.535003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317364.535066:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317364.535070:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317364.535073:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317364.664027:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317364.726756:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317365.686926:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317365.750906:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317366.713492:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317366.774837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317367.734875:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317367.801065:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317368.761796:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317368.822812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317369.654809:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317369.654821:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317369.654824:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317369.654827:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317369.782900:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317369.846842:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317370.810733:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317370.874123:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317371.831317:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317371.894840:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317372.598831:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317347/real 1767317347] req@ffff9d1f91032d80 x1853163904586496/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317372 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317372.598868:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032d80 x1853163904586496/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317372 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317372.598887:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317372.598906:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317372.598910:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904586496:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317372.599132:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317372.599137:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317372.599148:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317372.599154:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2594 00000100:00080000:1.0:1767317372.599165:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317372.599172:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317372.599206:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1880 x1853163904586752/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767317372.599231:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317372.599236:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317372.599298:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904586752:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317372.855150:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317372.918847:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317373.878929:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317373.942822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317374.648215:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317349/real 1767317349] req@ffff9d1f918d3800 x1853163904586624/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317374.648251:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3800 x1853163904586624/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317374 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317374.648273:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317374.648304:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317374.648309:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904586624:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767317374.648407:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317374.648413:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317374.648415:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317374.648418:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767317374.648421:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317374.648426:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2596 00000100:00080000:3.0:1767317374.648436:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317374.648443:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767317374.648453:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767317374.648484:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70000 x1853163904586880/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767317374.649068:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904586880:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317374.904569:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317374.967253:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317375.927703:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317375.990795:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317376.951002:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317377.015532:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317377.655089:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317377.655100:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317377.655103:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317377.655106:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317377.974921:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317378.039591:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317379.001508:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317379.062843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317379.702897:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317379.702908:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317379.702910:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317379.702913:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317380.022942:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317380.086850:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317381.047194:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317381.110838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317382.070934:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317382.135660:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317383.094903:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317383.158823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317384.121282:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317384.182791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317385.014832:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317385.014844:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317385.014847:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317385.014849:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317385.143009:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317385.206804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317386.168014:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317386.230800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317387.191079:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317387.254846:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317388.215665:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317388.278887:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317389.243935:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317389.303512:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317390.135308:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317390.135320:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317390.135323:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317390.135344:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317390.265291:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317390.326813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317391.286869:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317391.350896:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317392.312033:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317392.374819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317393.335361:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317393.398849:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317394.359080:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317394.423003:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317395.254796:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317395.254806:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317395.254809:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317395.254812:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317395.383638:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317395.447068:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317396.407554:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317396.470826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317397.110846:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317372/real 1767317372] req@ffff9d1f918d1880 x1853163904586752/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317397.110961:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1880 x1853163904586752/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317397 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317397.110984:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317397.111144:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317397.111162:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904586752:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767317397.115177:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317397.115187:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767317397.115191:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767317397.115201:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2620 00000100:00080000:0.0:1767317397.115213:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767317397.115219:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767317397.115258:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030000 x1853163904587008/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767317397.115287:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317397.115301:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317397.115649:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904587008:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317397.432673:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317397.494791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317398.455570:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317398.518802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317399.160721:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317374/real 1767317374] req@ffff9d1f92f70000 x1853163904586880/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317399 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317399.160758:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70000 x1853163904586880/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317399 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317399.160779:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317399.160795:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317399.160800:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904586880:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767317399.160909:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317399.160914:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317399.160917:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317399.160938:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767317399.160940:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767317399.160946:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2622 00000100:00080000:0.0:1767317399.160966:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767317399.160973:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767317399.160985:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767317399.161014:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031180 x1853163904587136/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767317399.161160:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904587136:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317399.479302:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317399.544670:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317400.502972:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317400.566799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317401.527217:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317401.590842:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317402.166772:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317402.166782:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317402.166785:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317402.166788:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317402.553155:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317402.619862:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317403.575943:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317403.639666:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317404.215296:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317404.215318:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317404.215321:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317404.215324:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317404.598983:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317404.662807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317405.622970:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317405.687144:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317406.646991:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317406.711716:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317407.670910:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317407.735339:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317408.694926:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317408.759873:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317409.590744:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317409.590753:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317409.590756:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317409.590760:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317409.719268:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317409.782867:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317410.743623:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317410.806796:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317411.766919:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317411.830861:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317412.796097:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317412.855034:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317413.815478:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317413.879955:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317414.710954:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317414.710966:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317414.710970:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317414.710972:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317414.839024:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317414.902802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317415.862961:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317415.926877:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317416.886998:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317416.950811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317417.911044:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317417.974982:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317418.936770:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317418.999309:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317419.832027:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317419.832037:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317419.832041:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317419.832067:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317419.959218:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317420.022823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317420.983026:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317421.046802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317422.007002:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317422.071639:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767317422.646802:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317397/real 1767317397] req@ffff9d1f91030000 x1853163904587008/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317422 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767317422.646836:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030000 x1853163904587008/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317422 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767317422.646856:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767317422.646889:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767317422.646893:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904587008:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767317422.647001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317422.647006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767317422.647009:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767317422.647014:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2644 00000100:00080000:0.0:1767317422.647022:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767317422.647028:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767317422.647056:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032300 x1853163904587264/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767317422.647087:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317422.647091:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1767317422.647288:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904587264:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317423.035669:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317423.100350:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317424.055111:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317424.118823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767317424.695012:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317399/real 1767317399] req@ffff9d1f91031180 x1853163904587136/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317424 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767317424.695058:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031180 x1853163904587136/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317424 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767317424.695081:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767317424.695121:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767317424.695126:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904587136:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767317424.695240:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317424.695246:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317424.695249:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317424.695252:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767317424.695255:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767317424.695261:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2646 00000100:00080000:0.0:1767317424.695271:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767317424.695278:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767317424.695290:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767317424.695323:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030a80 x1853163904587392/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767317424.695421:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904587392:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317425.080173:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317425.143050:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317426.103226:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317426.167194:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317427.126883:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317427.195821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317427.702816:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317427.702828:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317427.702831:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317427.702835:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317428.156514:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317428.214816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317429.180147:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317429.239906:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317429.751653:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317429.751665:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317429.751703:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317429.751707:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317430.199653:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317430.262858:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317431.223817:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317431.286978:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317432.248261:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317432.310828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317433.278161:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317433.334809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317434.295439:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317434.358777:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317435.191741:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317435.191752:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317435.191755:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317435.191758:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317435.318994:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317435.384119:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317436.344543:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317436.407857:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317437.368520:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317437.432527:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317438.390982:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317438.456359:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317439.417960:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317439.478800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317440.310976:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317440.310987:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317440.310990:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317440.310993:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317440.438939:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317440.502847:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317441.466471:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317441.526883:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317442.490458:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317442.551452:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317443.513706:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317443.574990:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317444.536389:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317444.598815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317445.430833:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317445.430843:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317445.430845:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317445.430848:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317445.561101:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317445.623352:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317446.585703:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317446.646987:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317447.160940:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317422/real 1767317422] req@ffff9d1f91032300 x1853163904587264/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317447 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317447.160979:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032300 x1853163904587264/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317447 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317447.161000:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317447.161030:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317447.161034:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904587264:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767317447.161305:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317447.161311:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767317447.161314:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317447.161335:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2670 00000100:00080000:3.0:1767317447.161344:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317447.161351:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767317447.161379:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73b80 x1853163904587520/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767317447.161406:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317447.161410:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317447.161465:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904587520:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317447.611808:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317447.671000:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317448.630886:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317448.696079:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317449.207721:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317424/real 1767317424] req@ffff9d1f91030a80 x1853163904587392/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317449.207764:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030a80 x1853163904587392/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317449 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317449.207784:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317449.207808:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317449.207812:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904587392:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767317449.207869:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317449.207875:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317449.207877:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317449.207881:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767317449.207884:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317449.207889:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2672 00000100:00080000:3.0:1767317449.207898:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317449.207905:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767317449.207916:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767317449.207946:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71880 x1853163904587648/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767317449.208020:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904587648:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317449.662951:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317449.718802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317450.679089:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317450.742903:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317451.703054:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317451.768080:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317452.215021:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317452.215039:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317452.215042:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317452.215065:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317452.727022:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317452.791535:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317453.750937:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317453.815067:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317454.263106:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317454.263118:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317454.263141:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317454.263145:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317454.776926:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317454.839114:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317455.798914:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317455.862801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317456.824705:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317456.886900:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317457.847250:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317457.910901:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317458.871241:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317458.934837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317459.770757:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317459.770768:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317459.770771:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317459.770774:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317459.895003:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317459.958849:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317460.919160:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317460.983358:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317461.943830:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317462.006922:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317462.967894:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317463.031031:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317463.992551:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317464.054866:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317464.889519:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317464.889528:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317464.889531:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317464.889534:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317465.015163:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317465.078857:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317466.049713:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317466.102898:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317467.062899:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317467.127215:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317468.089079:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317468.151731:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317469.110895:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317469.174913:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317470.007009:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317470.007017:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317470.007019:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317470.007022:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317470.134942:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317470.199268:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317471.166145:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317471.222793:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317472.182984:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317472.246820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317472.696037:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317447/real 1767317447] req@ffff9d1f92f73b80 x1853163904587520/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317472 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317472.696072:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73b80 x1853163904587520/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317472 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317472.696093:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317472.696124:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317472.696129:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904587520:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767317472.696922:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317472.696928:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767317472.696931:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767317472.696937:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2694 00000100:00080000:0.0:1767317472.696947:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767317472.696954:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767317472.696990:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031500 x1853163904587776/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767317472.697017:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317472.697021:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317472.697089:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904587776:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317473.210653:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317473.276512:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317474.230920:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317474.295057:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317474.743243:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317449/real 1767317449] req@ffff9d1f92f71880 x1853163904587648/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317474 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317474.743277:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71880 x1853163904587648/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317474 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317474.743297:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317474.743325:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317474.743330:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904587648:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317474.743481:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317474.743497:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317474.743500:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317474.743504:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317474.743507:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317474.743512:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2696 00000100:00080000:1.0:1767317474.743544:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317474.743551:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317474.743562:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317474.743588:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1c00 x1853163904587904/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767317474.746754:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904587904:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317475.254895:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317475.320171:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317476.278936:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317476.342837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317477.305754:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317477.366814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317477.750936:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317477.750946:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317477.750949:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317477.750953:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317478.327135:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317478.390809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317479.351001:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317479.414823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317479.798798:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317479.798810:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317479.798829:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317479.798833:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317480.376548:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317480.438824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317481.398924:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317481.462800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317482.423584:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317482.486801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317483.447393:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317483.510807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317484.471523:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317484.534799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317484.854890:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317484.854900:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317484.854903:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317484.854907:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317485.494926:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317485.558802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317486.521329:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317486.582934:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317487.543059:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317487.606810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317488.568553:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317488.630903:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317489.595008:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317489.654905:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317489.975306:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317489.975321:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317489.975325:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317489.975328:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317490.615006:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317490.678855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317491.639309:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317491.702858:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317492.662869:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317492.726780:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317493.687467:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317493.750845:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317494.711580:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317494.776183:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317495.094867:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317495.094881:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317495.094885:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317495.094889:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317495.735346:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317495.798923:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317496.759472:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317496.822901:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767317497.207400:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317472/real 1767317472] req@ffff9d1f91031500 x1853163904587776/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767317497.207451:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031500 x1853163904587776/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317497 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767317497.207477:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767317497.207582:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767317497.207588:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904587776:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317497.216067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317497.216083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317497.216089:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317497.216103:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2720 00000100:00080000:1.0:1767317497.216120:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317497.216128:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317497.216194:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0700 x1853163904588032/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767317497.216239:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317497.216243:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1767317497.217039:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904588032:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317497.784048:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317497.846936:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317498.808439:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317498.870873:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767317499.254807:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317474/real 1767317474] req@ffff9d1f918d1c00 x1853163904587904/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317499 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767317499.254839:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1c00 x1853163904587904/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317499 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767317499.254861:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767317499.254887:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767317499.254891:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904587904:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767317499.254974:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317499.254985:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317499.254989:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317499.254994:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767317499.254998:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767317499.255009:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2722 00000100:00080000:0.0:1767317499.255022:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767317499.255056:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767317499.255071:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767317499.255110:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032680 x1853163904588160/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767317499.255212:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904588160:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317499.831439:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317499.895040:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317500.855018:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317500.918801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317501.879120:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317501.943665:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317502.263124:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317502.263134:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317502.263154:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317502.263157:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317502.903331:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317502.966841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317503.927219:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317503.990837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317504.311113:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317504.311123:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317504.311126:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317504.311129:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317504.950915:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317505.014809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317505.974907:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317506.038801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317506.998954:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317507.063416:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317508.023002:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317508.086947:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317509.047859:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317509.110787:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317509.430807:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317509.430817:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317509.430819:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317509.430822:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317510.071723:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317510.134961:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317511.095098:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317511.158807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317512.118934:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317512.186588:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317513.142900:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317513.206852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317514.167719:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317514.230818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317514.550821:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317514.550832:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317514.550835:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317514.550838:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317515.191014:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317515.254804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317516.215086:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317516.278803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317517.239097:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317517.302810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317518.262936:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317518.326859:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317519.290029:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317519.351290:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317519.670946:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317519.670959:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317519.670961:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317519.670983:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317520.310926:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317520.374812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317521.334847:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317521.398821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317522.359003:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317522.422810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317522.743202:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317497/real 1767317497] req@ffff9d1f918d0700 x1853163904588032/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317522.743252:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0700 x1853163904588032/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317522 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317522.743276:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317522.743303:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317522.743308:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904588032:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767317522.743616:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317522.743622:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767317522.743625:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317522.743630:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2744 00000100:00080000:3.0:1767317522.743639:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317522.743644:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767317522.743714:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71880 x1853163904588288/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767317522.743740:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317522.743745:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317522.748744:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904588288:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317523.384811:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317523.446826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317524.408314:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317524.471059:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317524.794451:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317499/real 1767317499] req@ffff9d1f91032680 x1853163904588160/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317524.794496:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032680 x1853163904588160/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317524 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317524.794515:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317524.794540:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317524.794544:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904588160:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767317524.797646:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317524.797656:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317524.797660:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317524.797664:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767317524.797669:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767317524.797708:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2746 00000100:00080000:0.0:1767317524.797721:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767317524.797728:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767317524.797739:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767317524.797776:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032680 x1853163904588416/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767317524.798474:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904588416:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317525.431753:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317525.495849:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317526.456588:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317526.518874:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317527.479030:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317527.542975:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317527.799006:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317527.799017:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317527.799020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317527.799039:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317528.504821:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317528.566807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317529.531595:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317529.590880:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317529.847548:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317529.847559:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317529.847580:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317529.847584:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317530.552235:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317530.614830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317531.577548:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317531.639508:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317532.599710:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317532.662803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317533.622889:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317533.686817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317534.647209:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317534.714878:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317535.034273:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317535.034284:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317535.034287:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317535.034290:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317535.671889:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317535.734812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317536.695082:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317536.758798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317537.718907:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317537.783290:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317538.744058:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317538.806824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317539.768705:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317539.830838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317540.150960:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317540.150980:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317540.150984:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317540.150987:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317540.799712:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317540.854801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317541.815791:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317541.878794:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317542.839221:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317542.903766:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317543.863001:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317543.927162:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317544.887650:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317544.952472:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317545.272882:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317545.272901:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317545.272904:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317545.272910:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317545.911157:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317545.975071:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317546.935349:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317546.998922:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317547.261213:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317522/real 1767317522] req@ffff9d1f92f71880 x1853163904588288/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317547 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317547.261348:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71880 x1853163904588288/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317547 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317547.261386:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317547.261414:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317547.261418:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904588288:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767317547.263043:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317547.263052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767317547.263056:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317547.263076:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2770 00000100:00080000:2.0:1767317547.263088:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317547.263103:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767317547.263165:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cc380 x1853163904588544/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767317547.263220:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317547.263236:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317547.264980:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904588544:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317547.961668:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317548.022885:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317548.982884:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317549.046916:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317549.303129:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317524/real 1767317524] req@ffff9d1f91032680 x1853163904588416/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317549 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317549.303163:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032680 x1853163904588416/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317549 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317549.303183:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317549.303213:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317549.303217:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904588416:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767317549.303351:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317549.303356:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317549.303359:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317549.303363:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767317549.303366:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317549.303371:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2772 00000100:00080000:2.0:1767317549.303398:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317549.303406:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767317549.303419:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767317549.303449:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cd180 x1853163904588672/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317549.303580:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904588672:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317550.008501:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317550.070863:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317551.030947:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317551.094843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317552.055049:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317552.118824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317552.310981:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317552.310989:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317552.310991:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317552.310993:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317553.079101:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317553.142844:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317554.102931:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317554.166815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317554.359181:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317554.359193:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317554.359195:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317554.359199:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317555.128413:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317555.190834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317556.151378:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317556.215080:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317557.181318:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317557.239424:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317558.201516:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317558.266426:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317559.223006:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317559.288352:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317559.606796:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317559.606807:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317559.606810:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317559.606813:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317560.247714:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317560.310973:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317561.271271:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317561.334861:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317562.295664:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317562.358759:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317563.321557:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317563.384363:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317564.342951:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317564.406812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317564.727337:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317564.727348:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317564.727351:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317564.727354:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317565.366943:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317565.431108:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317566.393760:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317566.455094:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317567.414941:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317567.478820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317568.439406:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317568.503405:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317569.463360:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317569.526874:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317569.848892:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317569.848903:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317569.848906:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317569.848910:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317570.487396:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317570.550836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317571.511026:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317571.574808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317572.534948:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317572.598880:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317572.790890:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317547/real 1767317547] req@ffff9d1f967cc380 x1853163904588544/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317572 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317572.790925:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cc380 x1853163904588544/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317572 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317572.790945:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317572.790969:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317572.790974:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904588544:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767317572.791071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317572.791077:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767317572.791079:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317572.791085:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2794 00000100:00080000:2.0:1767317572.791111:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317572.791118:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767317572.791152:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cfb80 x1853163904588800/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767317572.791183:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317572.791202:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317572.791409:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904588800:192.168.202.140@tcp:250:kworker.0 00010000:00010000:1.0:1767317573.559046:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317573.622817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317574.586353:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317574.647449:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317574.838831:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317549/real 1767317549] req@ffff9d1f967cd180 x1853163904588672/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317574.838865:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cd180 x1853163904588672/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317574 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317574.838885:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317574.838914:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317574.838918:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904588672:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767317574.839150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317574.839156:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317574.839159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317574.839163:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767317574.839166:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317574.839171:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2796 00000100:00080000:2.0:1767317574.839191:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317574.839198:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767317574.839209:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767317574.839236:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cd880 x1853163904588928/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317574.839482:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904588928:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317575.606969:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317575.670833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317576.630913:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317576.694814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317577.655016:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317577.718851:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317577.846927:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317577.846938:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317577.846941:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317577.846944:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317578.679380:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317578.742802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317579.707152:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317579.767623:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317579.894941:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317579.894952:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317579.894955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317579.894959:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317580.727743:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317580.791717:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317581.752760:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317581.814827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317582.775062:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317582.839121:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317583.798898:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317583.862813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317584.822925:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317584.886905:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317585.207026:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317585.207038:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317585.207041:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317585.207044:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317585.847958:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317585.912054:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317586.873081:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317586.934925:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317587.895137:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317587.958834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317588.918904:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317588.983885:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317589.946831:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317590.010791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317590.327820:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317590.327836:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317590.327868:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317590.327872:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317590.966993:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317591.031320:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317591.991295:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317592.058203:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317593.015033:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317593.079252:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317594.039705:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317594.102921:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317595.064274:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317595.129645:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317595.446941:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317595.446957:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317595.446961:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317595.446965:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317596.087000:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317596.150816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317597.111090:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767317597.174963:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317597.306707:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317572/real 1767317572] req@ffff9d1f967cfb80 x1853163904588800/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317597.306747:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cfb80 x1853163904588800/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317597 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317597.306767:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317597.306792:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317597.306795:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904588800:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767317597.307617:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317597.307626:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767317597.307630:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317597.307648:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2820 00000100:00080000:3.0:1767317597.307661:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317597.307670:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767317597.307751:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73b80 x1853163904589056/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767317597.307782:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317597.307786:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317597.307997:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904589056:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317598.143990:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317598.201110:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317599.159272:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317599.223606:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317599.351885:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317574/real 1767317574] req@ffff9d1f967cd880 x1853163904588928/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317599.351933:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cd880 x1853163904588928/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317599 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317599.351963:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317599.351994:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317599.352000:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904588928:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767317599.352115:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317599.352123:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317599.352125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317599.352129:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767317599.352606:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317599.352615:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2822 00000100:00080000:2.0:1767317599.352628:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317599.352639:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767317599.352658:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767317599.352884:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cf800 x1853163904589184/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317599.357326:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904589184:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317600.184644:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317600.247613:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317601.207530:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317601.270829:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317601.398982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317601.398994:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317601.398997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317601.399000:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317602.231004:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317602.294879:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317603.255174:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317603.321139:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317604.279242:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317604.343847:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317604.407112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317604.407128:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317604.407131:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317604.407134:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317605.303813:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317605.366895:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317606.328260:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317606.395567:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317607.350981:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317607.414887:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317608.375488:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317608.438983:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317609.400417:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317609.465270:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317609.784416:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317609.784431:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317609.784435:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317609.784439:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317610.425656:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317610.487110:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317611.447019:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317611.511664:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317612.471640:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317612.534874:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317613.494981:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317613.561036:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317614.524462:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317614.589194:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317614.903041:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317614.903056:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317614.903060:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317614.903063:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317615.544113:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317615.607392:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317616.568439:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317616.631592:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317617.590904:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317617.655256:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317618.614897:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317618.679702:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317619.639167:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317619.703360:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317620.024132:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317620.024139:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317620.024142:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317620.024145:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317620.669440:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317620.726828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317621.686896:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317621.752053:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317622.713977:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317622.774847:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317622.839560:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317597/real 1767317597] req@ffff9d1f92f73b80 x1853163904589056/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317622.839595:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73b80 x1853163904589056/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317622.839615:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317622.839655:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317622.839660:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904589056:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767317622.839759:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317622.839765:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767317622.839786:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317622.839791:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2844 00000100:00080000:3.0:1767317622.839801:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317622.839808:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767317622.839839:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70000 x1853163904589312/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767317622.839864:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317622.839869:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317622.840133:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904589312:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317623.735290:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317623.799018:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317624.760728:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317624.822872:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767317624.887877:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317599/real 1767317599] req@ffff9d1f967cf800 x1853163904589184/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767317624.887914:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cf800 x1853163904589184/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317624 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767317624.887932:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767317624.887970:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767317624.887975:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904589184:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767317624.888515:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317624.888521:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317624.888523:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317624.888527:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767317624.888531:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317624.888537:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2847 00000100:00080000:3.0:1767317624.888547:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317624.888554:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767317624.888566:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767317624.888599:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71500 x1853163904589440/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767317624.888884:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904589440:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317625.786521:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317625.847228:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317626.807111:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317626.871005:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317627.830976:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317627.894894:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317627.895101:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317627.895123:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317627.895126:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317627.895129:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317628.855384:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317628.919012:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317629.879489:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1767317629.942984:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317629.942995:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317629.942998:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317629.943002:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767317629.943176:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317630.902893:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317630.969052:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317631.927619:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317631.991030:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317632.952200:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317633.015398:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317633.975032:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317634.038877:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317634.998977:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317635.062815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317635.382992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317635.383002:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317635.383005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317635.383008:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317636.025720:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317636.088041:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317637.046926:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317637.111207:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317638.073061:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317638.140288:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317639.094938:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317639.160435:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317640.121947:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317640.183116:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317640.508717:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317640.508729:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317640.508731:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317640.508735:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317641.143279:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317641.207074:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317642.166921:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317642.234211:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317643.191965:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317643.256019:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317644.215129:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317644.278802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317645.239092:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317645.302895:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317645.622920:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317645.622930:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317645.622932:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317645.622934:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317646.263088:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317646.327437:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317647.287121:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1767317647.350989:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317622/real 1767317622] req@ffff9d1f92f70000 x1853163904589312/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317647 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317647.351034:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70000 x1853163904589312/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317647 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317647.351063:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317647.351088:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317647.351111:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904589312:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317647.351182:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317647.351187:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317647.351190:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317647.351196:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2870 00000100:00080000:1.0:1767317647.351205:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317647.351212:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317647.351243:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3480 x1853163904589568/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767317647.351268:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317647.351272:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317647.351510:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904589568:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767317647.360648:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317648.311129:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317648.375391:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317649.336057:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1767317649.398851:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317624/real 1767317624] req@ffff9d1f92f71500 x1853163904589440/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317649.398887:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71500 x1853163904589440/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317649 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317649.398907:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317649.398933:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317649.398937:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904589440:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767317649.399065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317649.399070:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317649.399072:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317649.399075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767317649.399078:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317649.399083:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2872 00000100:00080000:2.0:1767317649.399093:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317649.399099:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767317649.399129:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767317649.399157:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cf800 x1853163904589696/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317649.399245:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904589696:192.168.202.140@tcp:38:kworker.0 00000100:00100000:3.0:1767317649.399820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317650.358998:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317650.422934:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317651.383197:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317651.446846:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317652.415112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317652.415122:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317652.415125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317652.415128:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317652.416373:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317652.470872:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317653.431008:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317653.494889:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317654.455119:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1767317654.455224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317654.455231:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317654.455233:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317654.455237:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1767317654.518794:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317655.482624:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317655.545145:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317656.517794:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317656.567273:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317657.526988:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317657.590849:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317658.550903:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317658.614833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317659.574857:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317659.638812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317659.959434:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317659.959445:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317659.959448:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317659.959450:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317660.601547:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317660.662754:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317661.623205:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317661.687538:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317662.647462:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317662.710824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317663.670882:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317663.734802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317664.695092:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317664.758994:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317665.078974:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317665.078988:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317665.078992:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317665.078996:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317665.727480:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317665.783459:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317666.744931:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317666.808447:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317667.768014:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317667.834537:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317668.794422:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317668.855209:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317669.815406:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317669.878801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317670.198888:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317670.198903:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317670.198906:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317670.198910:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317670.842595:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317670.902812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317671.863009:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317671.926789:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317672.886851:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317647/real 1767317647] req@ffff9d1f918d3480 x1853163904589568/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317672 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317672.886894:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3480 x1853163904589568/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317672 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317672.886916:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317672.886944:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317672.886948:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904589568:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317672.887064:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317672.887070:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317672.887079:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317672.887084:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2895 00000100:00080000:1.0:1767317672.887092:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317672.887099:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317672.887142:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1180 x1853163904589824/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767317672.887166:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317672.887169:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317672.887617:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904589824:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317672.887910:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317672.950881:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317673.920382:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317673.974826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317674.934865:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317649/real 1767317649] req@ffff9d1f967cf800 x1853163904589696/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317674.934899:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cf800 x1853163904589696/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317674 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317674.934918:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317674.935092:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317674.935097:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904589696:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317674.939206:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1767317674.943026:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317674.943039:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317674.943043:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317674.943046:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317674.943052:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317674.943063:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2897 00000100:00080000:1.0:1767317674.943075:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317674.943082:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317674.943096:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317674.943137:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1500 x1853163904589952/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317674.943462:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904589952:192.168.202.140@tcp:38:kworker.0 00000100:00100000:3.0:1767317674.998825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317675.959013:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317676.022861:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317676.985802:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317677.046820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317678.007701:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1767317678.008606:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317678.008613:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317678.008616:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317678.008619:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1767317678.072484:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317679.032411:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317679.094811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317680.054857:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317680.054866:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317680.054869:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317680.054873:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317680.055079:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317680.121873:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317681.079043:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317681.142997:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317682.102921:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317682.166841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317683.127067:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317683.190877:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317684.150987:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317684.214832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317685.174964:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317685.238843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317685.559464:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317685.559475:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317685.559478:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317685.559481:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317686.201498:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317686.263644:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317687.223254:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317687.287398:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317688.247285:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317688.310874:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317689.270898:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317689.334762:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317690.295209:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317690.359075:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317690.684286:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317690.684296:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317690.684299:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317690.684303:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317691.318875:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317691.382806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317692.343263:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317692.406831:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317693.366944:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317693.431514:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317694.390927:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317694.455991:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317695.414881:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317695.479729:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317695.799361:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317695.799378:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317695.799381:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317695.799385:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317696.440977:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317696.504905:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317697.463336:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1767317697.463499:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317672/real 1767317672] req@ffff9d1f918d1180 x1853163904589824/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317697.463541:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1180 x1853163904589824/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317697 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317697.463562:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317697.463610:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317697.463615:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904589824:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767317697.463710:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317697.463715:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767317697.463718:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317697.463724:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2920 00000100:00080000:3.0:1767317697.463733:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317697.463739:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767317697.463775:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72a00 x1853163904590080/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767317697.463801:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317697.463805:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317697.464136:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904590080:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767317697.527147:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317698.487945:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317698.550927:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317699.510832:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317674/real 1767317674] req@ffff9d1f918d1500 x1853163904589952/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317699 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317699.510870:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1500 x1853163904589952/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317699 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317699.510891:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317699.510922:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317699.510927:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904589952:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317699.511162:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1767317699.511476:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317699.511484:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317699.511486:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317699.511490:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317699.511493:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317699.511500:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2922 00000100:00080000:1.0:1767317699.511509:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317699.511517:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317699.511527:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317699.511577:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d2300 x1853163904590208/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317699.512876:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904590208:192.168.202.140@tcp:38:kworker.0 00000100:00100000:3.0:1767317699.574806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317700.536085:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317700.600336:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317701.565587:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317701.623918:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317702.518811:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317702.518823:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317702.518853:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317702.518857:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317702.583980:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317702.646811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317703.610618:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317703.672218:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317704.566876:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317704.566887:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317704.566890:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317704.566894:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317704.635775:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317704.694800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317705.663236:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317705.718892:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317706.678920:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317706.742848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317707.703218:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317707.766796:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317708.727425:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317708.791303:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317709.624018:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317709.624028:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317709.624031:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317709.624034:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317709.750878:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317709.814854:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317710.776656:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317710.838813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317711.801720:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317711.862813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317712.822918:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317712.886807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317713.846931:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317713.911804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317714.743031:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317714.743063:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317714.743066:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317714.743069:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317714.871042:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317714.935105:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317715.894954:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317715.958812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317716.919285:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317716.982953:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317717.942925:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317718.007964:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317718.970596:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317719.030913:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317719.862841:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317719.862851:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317719.862853:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317719.862856:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317719.993254:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317720.055439:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317721.015124:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317721.078811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317722.040498:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317722.102827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317722.999865:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317697/real 1767317697] req@ffff9d1f92f72a00 x1853163904590080/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317722.999902:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72a00 x1853163904590080/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317722 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317722.999925:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317722.999958:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317722.999963:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904590080:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317723.000061:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317723.000066:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317723.000070:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317723.000075:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2945 00000100:00080000:1.0:1767317723.000084:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317723.000090:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317723.000122:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0380 x1853163904590336/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767317723.000148:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317723.000153:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317723.000709:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904590336:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317723.064344:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317723.126960:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317724.024157:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317699/real 1767317699] req@ffff9d1f918d2300 x1853163904590208/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317724.024190:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d2300 x1853163904590208/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317724 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317724.024211:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317724.024243:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317724.024247:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904590208:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317724.024405:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317724.024411:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317724.024414:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317724.024417:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317724.024421:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317724.024426:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2947 00000100:00080000:1.0:1767317724.024436:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317724.024457:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317724.024468:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317724.024496:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3b80 x1853163904590464/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317724.024587:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904590464:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317724.090246:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317724.150812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317725.112493:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317725.175006:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317726.134891:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317726.199657:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317727.159263:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317727.222796:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317728.055014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317728.055024:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317728.055027:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317728.055030:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317728.187875:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317728.246802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317729.079084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317729.079111:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317729.079114:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317729.079118:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317729.207767:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317729.273531:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317730.231322:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317730.294810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317731.255076:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317731.318797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317732.279485:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317732.342990:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317733.303313:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317733.367439:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317734.199973:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317734.199995:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317734.199999:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317734.200001:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317734.327269:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317734.394158:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317735.351463:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317735.416361:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317736.375332:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317736.439457:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317737.398917:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317737.462809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317738.422876:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317738.486875:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317739.319030:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317739.319041:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317739.319044:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317739.319047:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317739.450918:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317739.510905:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317740.473378:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317740.535467:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317741.495352:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317741.562322:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317742.519482:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317742.582780:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317743.546073:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317743.611525:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317744.439040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317744.439050:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317744.439060:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317744.439064:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317744.567121:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317744.633011:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317745.592266:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317745.654834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317746.615045:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317746.678824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317747.640612:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317747.702799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317748.534819:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317723/real 1767317723] req@ffff9d1f918d0380 x1853163904590336/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317748.534857:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0380 x1853163904590336/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317748 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317748.534877:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317748.534900:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317748.534905:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904590336:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317748.534936:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317748.534942:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317748.534945:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317748.534950:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2970 00000100:00080000:1.0:1767317748.534960:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317748.534966:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317748.535014:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0380 x1853163904590592/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767317748.535040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317748.535045:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317748.535077:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904590592:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317748.663828:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317748.726812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317749.558906:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317724/real 1767317724] req@ffff9d1f918d3b80 x1853163904590464/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317749.558954:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3b80 x1853163904590464/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317749 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317749.558974:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317749.559001:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317749.559005:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904590464:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317749.559103:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317749.559108:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317749.559111:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317749.559145:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317749.559149:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317749.559154:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2971 00000100:00080000:1.0:1767317749.559164:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317749.559170:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317749.559188:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317749.559214:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d2300 x1853163904590720/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317749.559291:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904590720:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767317749.687169:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317749.751289:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317750.711030:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317750.774808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317751.735123:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317751.805938:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317752.759475:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317752.823467:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317753.590887:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317753.590898:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317753.590901:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317753.590903:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317753.782915:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317753.847333:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317754.614785:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317754.614810:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317754.614813:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317754.614816:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317754.808936:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317754.870810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317755.831114:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317755.894818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317756.855076:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317756.918926:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317757.879514:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317757.943175:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317758.904112:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317758.967177:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317759.799130:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317759.799143:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317759.799147:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317759.799151:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317759.928519:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317759.992591:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317760.952221:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317761.016145:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317761.976397:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317762.039210:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317763.004617:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317763.062936:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317764.023069:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317764.086911:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317764.918924:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317764.918934:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317764.918937:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317764.918941:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317765.047333:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317765.110931:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317766.078558:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317766.134879:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317767.097599:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317767.158845:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317768.120166:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317768.182805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317769.143148:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317769.206843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317770.038813:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317770.038824:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317770.038827:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317770.038831:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317770.168067:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317770.231876:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317771.190941:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317771.254798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317772.215005:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317772.278807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317773.047073:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317748/real 1767317748] req@ffff9d1f918d0380 x1853163904590592/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317773 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317773.047100:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0380 x1853163904590592/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317773 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317773.047117:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317773.047140:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317773.047144:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904590592:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767317773.047185:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317773.047191:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767317773.047194:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317773.047200:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 2996 00000100:00080000:2.0:1767317773.047210:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317773.047244:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767317773.047278:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967ce680 x1853163904590848/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767317773.047307:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317773.047312:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317773.047418:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904590848:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317773.239817:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317773.302803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317774.071007:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317749/real 1767317749] req@ffff9d1f918d2300 x1853163904590720/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317774 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317774.071048:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d2300 x1853163904590720/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317774 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317774.071070:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317774.071095:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317774.071098:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904590720:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767317774.071392:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317774.071397:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317774.071406:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317774.071410:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767317774.071413:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317774.071417:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 2997 00000100:00080000:2.0:1767317774.071426:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317774.071433:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767317774.071444:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767317774.071472:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cf480 x1853163904590976/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317774.072669:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904590976:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317774.262996:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317774.326815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317775.287235:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317775.354499:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317776.310930:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317776.374805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317777.337468:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317777.400534:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317778.103448:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317778.103459:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317778.103461:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317778.103464:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317778.359004:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317778.422959:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317779.130136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317779.130147:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317779.130151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317779.130154:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317779.385217:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317779.450113:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317780.407183:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317780.472371:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317781.430919:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317781.498154:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317782.461170:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317782.519404:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317783.478967:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317783.542801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317784.378172:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317784.378182:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317784.378184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317784.378187:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317784.503035:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317784.566881:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317785.527731:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317785.591257:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317786.550878:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317786.614837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317787.574926:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317787.638797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317788.600021:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317788.664250:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317789.495003:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317789.495015:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317789.495018:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317789.495021:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317789.623121:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317789.686799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317790.646987:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317790.710842:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317791.670909:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317791.735207:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317792.695023:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317792.759162:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317793.719078:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317793.782810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317794.614854:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317794.614866:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317794.614869:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317794.614873:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317794.746199:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317794.806813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317795.767063:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317795.830846:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317796.790942:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317796.855186:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317797.814945:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317797.880410:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317798.582862:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317773/real 1767317773] req@ffff9d1f967ce680 x1853163904590848/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317798.582899:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967ce680 x1853163904590848/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317798 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317798.582922:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317798.582946:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317798.582950:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904590848:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767317798.585870:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317798.585881:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767317798.585885:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317798.585894:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3020 00000100:00080000:3.0:1767317798.585905:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317798.585913:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767317798.585957:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72d80 x1853163904591104/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767317798.585997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317798.586002:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317798.586046:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904591104:192.168.202.140@tcp:250:kworker.0 00010000:00010000:1.0:1767317798.839006:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317798.903089:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317799.606937:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317774/real 1767317774] req@ffff9d1f967cf480 x1853163904590976/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317799.606986:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cf480 x1853163904590976/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317799 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317799.607011:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317799.607047:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317799.607052:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904590976:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767317799.607450:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317799.607456:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317799.607458:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317799.607472:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767317799.607476:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317799.607501:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3021 00000100:00080000:2.0:1767317799.607510:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317799.607516:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767317799.607528:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767317799.607557:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cf480 x1853163904591232/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317799.608638:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904591232:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317799.864524:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317799.926864:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317800.890731:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317800.950853:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317801.910881:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317801.975907:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317802.934910:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317802.999623:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317803.638935:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317803.638946:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317803.638949:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317803.638952:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317803.959048:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317804.022801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317804.663289:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317804.663300:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317804.663302:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317804.663323:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317804.983397:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317805.046852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317806.012740:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317806.070916:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317807.032567:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317807.094814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317808.054956:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317808.118808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317809.079041:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317809.142850:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317809.975083:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317809.975094:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317809.975097:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317809.975100:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317810.103993:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317810.168408:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317811.128371:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317811.191304:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317812.150945:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317812.215214:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317813.175169:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317813.239085:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317814.199175:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317814.262793:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317815.094854:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317815.094865:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317815.094868:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317815.094871:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317815.223300:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317815.286883:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317816.247310:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317816.310811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317817.271048:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317817.335847:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317818.296993:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317818.358882:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317819.318973:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317819.390010:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317820.214868:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317820.214877:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317820.214881:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317820.214884:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317820.343265:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317820.406799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317821.366900:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317821.433819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317822.390989:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317822.458826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317823.095054:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317798/real 1767317798] req@ffff9d1f92f72d80 x1853163904591104/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317823.095610:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72d80 x1853163904591104/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317823 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317823.095633:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317823.095661:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317823.095665:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904591104:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767317823.096125:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317823.096132:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767317823.096136:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317823.096142:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3046 00000100:00080000:2.0:1767317823.096154:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317823.096502:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767317823.096612:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967ce680 x1853163904591360/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767317823.096658:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317823.096662:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317823.097291:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904591360:192.168.202.140@tcp:250:kworker.0 00010000:00010000:1.0:1767317823.417666:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317823.478872:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317824.118854:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317799/real 1767317799] req@ffff9d1f967cf480 x1853163904591232/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317824.118919:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cf480 x1853163904591232/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317824 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317824.118941:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317824.118967:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317824.118972:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904591232:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767317824.119069:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317824.119075:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317824.119077:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317824.119081:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767317824.119083:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317824.119089:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3047 00000100:00080000:3.0:1767317824.119099:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317824.119106:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767317824.119119:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767317824.119156:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72680 x1853163904591488/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317824.119388:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904591488:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767317824.438906:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317824.502857:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317825.463946:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317825.526805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317826.486901:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317826.550915:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317827.515078:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317827.574815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317828.151418:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317828.151431:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317828.151465:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317828.151471:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317828.534906:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317828.598798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317829.174805:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317829.174813:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317829.174815:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317829.174818:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317829.559272:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317829.622984:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317830.587586:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317830.647188:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317831.606877:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317831.670815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317832.630991:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317832.695064:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317833.654856:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317833.720293:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317834.551091:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317834.551103:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317834.551105:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317834.551108:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317834.682165:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317834.743009:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317835.703006:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317835.766787:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317836.728236:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317836.790898:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317837.751577:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317837.814823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317838.777006:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317838.838830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317839.670975:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317839.670986:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317839.670989:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317839.670993:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317839.799321:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317839.865204:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317840.823891:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317840.886815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317841.846982:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317841.910892:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317842.871475:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317842.934780:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317843.900536:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317843.960231:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317844.791165:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317844.791180:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317844.791184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317844.791187:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317844.919412:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317844.982802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317845.943173:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317846.006889:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317846.968637:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317847.030806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317848.013903:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317848.056630:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317848.630814:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317823/real 1767317823] req@ffff9d1f967ce680 x1853163904591360/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317848 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317848.630846:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967ce680 x1853163904591360/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317848 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317848.630867:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317848.630895:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317848.630899:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904591360:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767317848.631000:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317848.631005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767317848.631009:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767317848.631015:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3070 00000100:00080000:2.0:1767317848.631024:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767317848.631031:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767317848.631082:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cf800 x1853163904591616/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317848.631143:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904591616:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767317848.631724:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317848.631729:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317849.014850:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317849.078798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317849.657611:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317824/real 1767317824] req@ffff9d1f92f72680 x1853163904591488/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317849 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317849.657643:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72680 x1853163904591488/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317849 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317849.657664:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317849.657705:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317849.657709:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904591488:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317849.657802:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317849.657808:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317849.657810:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317849.657813:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317849.657816:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317849.657821:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3071 00000100:00080000:1.0:1767317849.657830:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317849.657836:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317849.657847:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317849.657877:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1500 x1853163904591744/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317849.660354:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904591744:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317850.039447:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317850.102796:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317851.062991:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317851.126823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317852.086958:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317852.150822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317853.110930:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317853.174827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317853.690419:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317853.690451:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317853.690454:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317853.690457:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317854.135497:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317854.198850:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317854.712117:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317854.712267:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317854.712270:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317854.712273:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317855.161810:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317855.223568:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317856.182934:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317856.246890:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317857.207338:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317857.271276:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317858.234338:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317858.294834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317859.254942:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317859.319074:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317860.151015:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317860.151027:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317860.151030:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317860.151033:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317860.282407:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317860.342912:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317861.307978:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317861.367065:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317862.327036:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317862.392752:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317863.351168:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317863.414890:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317864.374922:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317864.440269:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317865.270888:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317865.270904:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317865.270907:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317865.270911:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317865.400204:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317865.468180:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317866.427126:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317866.486812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317867.447018:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317867.510820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317868.471294:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317868.539413:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317869.503184:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317869.558795:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317870.390822:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317870.390832:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317870.390835:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317870.390838:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317870.524779:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317870.608727:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317871.543102:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317871.674133:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317872.566899:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317872.699527:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317873.142837:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317848/real 1767317848] req@ffff9d1f967cf800 x1853163904591616/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317873 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317873.142893:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cf800 x1853163904591616/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317873 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317873.142913:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317873.142942:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317873.142947:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904591616:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767317873.143554:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317873.143560:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767317873.143563:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317873.143568:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3096 00000100:00080000:3.0:1767317873.143578:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317873.143584:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767317873.143613:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71f80 x1853163904591872/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767317873.143639:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317873.143643:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317873.143735:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904591872:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317873.590887:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317873.718822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317874.168621:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317849/real 1767317849] req@ffff9d1f918d1500 x1853163904591744/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317874 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317874.168659:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1500 x1853163904591744/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317874 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317874.168696:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317874.168722:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317874.168727:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904591744:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317874.168807:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317874.168812:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317874.168815:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317874.168818:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317874.168822:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317874.168827:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3097 00000100:00080000:1.0:1767317874.168839:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317874.168847:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317874.168857:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317874.168890:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1180 x1853163904592000/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317874.169421:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904592000:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317874.614905:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317874.743095:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317875.641264:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317875.770636:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317876.663378:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317876.790814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317877.687566:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317877.815303:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317878.199103:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317878.199117:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317878.199120:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317878.199124:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317878.711657:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317878.838908:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317879.223158:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317879.223168:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317879.223170:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317879.223191:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317879.735558:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317879.862843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317880.766543:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317880.886983:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317881.783009:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317881.911034:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317882.808609:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317882.937904:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317883.830999:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317883.958849:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317884.732542:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317884.732555:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317884.732558:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317884.732581:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317884.855150:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317884.984296:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317885.878895:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317886.007181:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317886.903845:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317887.032866:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317887.928250:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317888.056026:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317888.951193:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317889.078889:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317889.847024:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317889.847036:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317889.847039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317889.847043:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317889.974948:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317890.102825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317891.000094:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317891.128170:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317892.025582:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317892.150866:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317893.050857:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317893.174799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317894.070815:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317894.198835:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317894.966967:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317894.966979:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317894.966982:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317894.966986:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317895.094900:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317895.222814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317896.119098:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317896.248426:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317897.143434:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317897.270827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317898.168087:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317898.295545:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317898.678824:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317873/real 1767317873] req@ffff9d1f92f71f80 x1853163904591872/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317898 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317898.678879:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71f80 x1853163904591872/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317898 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317898.678900:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317898.678971:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317898.678976:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904591872:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767317898.680114:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317898.680121:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767317898.680124:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317898.680132:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3120 00000100:00080000:3.0:1767317898.680216:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317898.680223:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767317898.680259:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73480 x1853163904592128/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767317898.680289:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317898.680294:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317898.680423:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904592128:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317899.192567:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317899.320528:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317899.708585:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317874/real 1767317874] req@ffff9d1f918d1180 x1853163904592000/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317899 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317899.708614:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1180 x1853163904592000/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317899 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317899.708634:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317899.708660:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317899.708664:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904592000:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767317899.715731:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317899.715741:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317899.715772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317899.715776:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767317899.715782:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317899.715795:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3121 00000100:00080000:3.0:1767317899.715807:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317899.715814:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767317899.715828:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767317899.715868:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71180 x1853163904592256/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317899.719040:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904592256:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317900.214939:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317900.342827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317901.238897:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317901.366872:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317902.263632:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317902.390839:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317903.286925:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317903.415631:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317903.734894:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317903.734902:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317903.734905:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317903.734908:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317904.313005:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317904.438831:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317904.763229:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317904.763239:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317904.763242:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317904.763244:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317905.337024:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317905.462796:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317906.361238:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317906.486808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317907.382961:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317907.511190:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317908.407001:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317908.534905:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317909.430894:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317909.558801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317909.815197:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317909.815209:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317909.815212:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317909.815215:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317910.454993:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317910.582831:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317911.480268:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317911.606803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317912.505415:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317912.631719:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317913.527060:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317913.655190:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317914.551089:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317914.678941:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317914.934859:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317914.934869:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317914.934872:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317914.934875:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317915.574910:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317915.702827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317916.600093:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317916.726818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317917.622990:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317917.750897:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317918.647544:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317918.774892:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317919.670926:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317919.799736:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317920.057181:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317920.057195:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317920.057199:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317920.057202:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317920.696163:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317920.822819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317921.718923:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317921.847651:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317922.743249:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317922.872833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317923.190891:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317898/real 1767317898] req@ffff9d1f92f73480 x1853163904592128/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317923.190928:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73480 x1853163904592128/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317923 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317923.190949:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317923.190961:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317923.190965:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904592128:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767317923.191100:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317923.191106:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767317923.191109:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317923.191115:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3146 00000100:00080000:3.0:1767317923.191125:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317923.191132:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767317923.191169:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73800 x1853163904592384/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767317923.191192:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317923.191196:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317923.191253:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904592384:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767317923.767102:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317923.894826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317924.215452:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317899/real 1767317899] req@ffff9d1f92f71180 x1853163904592256/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317924 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317924.215490:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71180 x1853163904592256/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317924 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317924.215523:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317924.215560:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317924.215565:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904592256:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317924.215729:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317924.215735:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317924.215737:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317924.215741:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317924.215761:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317924.215766:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3147 00000100:00080000:1.0:1767317924.215775:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317924.215781:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317924.215793:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317924.215818:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3480 x1853163904592512/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317924.216064:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904592512:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767317924.796464:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317924.919889:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317925.821726:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317925.942820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317926.841260:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317926.966798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317927.863158:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317927.991034:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317928.247198:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317928.247206:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317928.247208:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317928.247211:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317928.887738:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317929.016042:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317929.271590:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317929.271604:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317929.271608:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317929.271611:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317929.910911:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317930.042859:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317930.935088:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317931.063240:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317931.958953:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317932.086853:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317932.984585:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317933.110824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317934.008350:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317934.134808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317934.390851:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317934.390862:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317934.390864:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317934.390867:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317935.031175:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317935.158989:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317936.054958:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317936.182815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317937.078937:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317937.207699:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317938.105777:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317938.230827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317939.127701:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317939.255315:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317939.510955:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317939.510965:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317939.510968:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317939.510971:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317940.150880:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317940.279977:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317941.177160:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317941.305527:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317942.198976:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317942.327704:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317943.223634:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317943.350796:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317944.248396:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317944.374810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317944.632307:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317944.632316:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317944.632319:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317944.632322:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317945.272997:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317945.400288:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317946.295019:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317946.422826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317947.320308:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317947.446809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317948.348562:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317948.471085:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317948.726957:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317923/real 1767317923] req@ffff9d1f92f73800 x1853163904592384/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317948.727007:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73800 x1853163904592384/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317948 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317948.727034:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317948.727072:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317948.727078:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904592384:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767317948.727287:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317948.727294:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767317948.727298:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767317948.727336:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3170 00000100:00080000:3.0:1767317948.727347:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767317948.727354:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767317948.727392:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70700 x1853163904592640/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767317948.727506:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317948.727511:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767317948.728000:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904592640:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317949.370799:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317949.495111:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317949.753362:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317924/real 1767317924] req@ffff9d1f918d3480 x1853163904592512/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317949.753416:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3480 x1853163904592512/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317949 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317949.753471:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317949.753486:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317949.753492:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904592512:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317949.753853:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317949.753861:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317949.753864:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317949.753869:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317949.753873:0:36:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317949.753881:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3171 00000100:00080000:1.0:1767317949.753895:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317949.753905:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317949.753921:0:36:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317949.753958:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0700 x1853163904592768/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767317949.754059:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904592768:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767317950.392047:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317950.518915:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317951.415133:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317951.544049:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317952.439295:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317952.568234:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317953.464667:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317953.590850:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317953.786233:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317953.786258:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317953.786270:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317953.786274:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317954.487700:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317954.614888:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317954.806913:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317954.806931:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317954.806935:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317954.806938:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317955.511289:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317955.639477:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317956.537637:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317956.663057:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317957.559586:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317957.686864:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317958.584952:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317958.711947:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317959.608188:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317959.734860:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767317959.993492:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317959.993507:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767317959.993511:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767317959.993515:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317960.630959:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317960.758920:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317961.655255:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317961.782923:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317962.680423:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317962.807634:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317963.703068:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317963.830755:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317964.731298:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317964.854818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317965.113143:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317965.113155:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317965.113166:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317965.113170:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317965.751661:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317965.878859:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317966.775419:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317966.902783:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317967.799434:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317967.927021:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317968.822912:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317968.952734:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317969.846979:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317969.974784:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317970.230858:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317970.230870:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317970.230891:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317970.230895:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317970.871067:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317971.000177:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317971.894981:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317972.022879:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317972.918940:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317973.047400:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317973.238860:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317948/real 1767317948] req@ffff9d1f92f70700 x1853163904592640/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317973.238907:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70700 x1853163904592640/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317973 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317973.238927:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317973.238937:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317973.238941:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904592640:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317973.239137:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317973.239142:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317973.239146:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317973.239152:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3196 00000100:00080000:1.0:1767317973.239161:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317973.239168:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317973.239195:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1c00 x1853163904592896/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767317973.239222:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317973.239226:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1767317973.239311:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904592896:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767317973.943061:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317974.070788:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767317974.263367:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317949/real 1767317949] req@ffff9d1f918d0700 x1853163904592768/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317974 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767317974.263416:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0700 x1853163904592768/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317974 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767317974.263439:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767317974.263554:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767317974.263559:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904592768:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767317974.266351:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317974.266360:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317974.266363:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317974.266367:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767317974.266371:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767317974.266451:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3197 00000100:00080000:0.0:1767317974.266462:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767317974.266468:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767317974.266529:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767317974.266562:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032680 x1853163904593024/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767317974.267301:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904593024:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767317974.968512:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317975.098296:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317975.990901:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317976.118955:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317977.014928:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317977.144254:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317978.038952:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317978.166787:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767317978.295140:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317978.295151:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767317978.295154:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767317978.295157:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317979.065646:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317979.191712:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317979.318845:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317979.318854:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317979.318856:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317979.318858:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317980.086853:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317980.214834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317981.118075:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317981.239137:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317982.134924:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317982.264289:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317983.159215:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317983.286862:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317984.184610:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317984.310817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767317984.566815:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317984.566827:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317984.566830:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317984.566854:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767317985.206893:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317985.337202:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317986.231461:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317986.359631:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317987.255510:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317987.382792:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317988.280594:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317988.406838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317989.302949:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317989.433577:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317989.687067:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317989.687077:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317989.687080:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317989.687083:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767317990.329883:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317990.455040:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767317991.351627:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317991.478863:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317992.375782:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317992.503127:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317993.399177:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767317993.530326:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317994.424975:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317994.552556:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767317994.807170:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317994.807181:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767317994.807184:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767317994.807187:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317995.448356:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317995.577248:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317996.472071:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317996.598805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767317997.494976:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317997.622803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767317998.518897:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317998.646833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317998.775014:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317973/real 1767317973] req@ffff9d1f918d1c00 x1853163904592896/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317998 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317998.775051:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1c00 x1853163904592896/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767317998 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317998.775074:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317998.775163:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317998.775168:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904592896:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317998.776402:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317998.776411:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767317998.776415:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317998.776425:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3220 00000100:00080000:1.0:1767317998.776436:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317998.776442:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767317998.776474:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1880 x1853163904593152/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317998.776548:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904593152:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767317998.776654:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317998.776658:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767317999.543714:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767317999.671072:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767317999.798914:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317974/real 1767317974] req@ffff9d1f91032680 x1853163904593024/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317999 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767317999.798979:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032680 x1853163904593024/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767317999 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767317999.799003:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767317999.799032:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767317999.799037:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904593024:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767317999.799371:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317999.799377:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767317999.799380:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767317999.799384:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767317999.799386:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767317999.799391:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3221 00000100:00080000:1.0:1767317999.799400:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767317999.799415:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767317999.799425:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767317999.799460:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3800 x1853163904593280/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767317999.799545:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904593280:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318000.567017:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318000.695044:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318001.590897:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318001.718825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318002.614982:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318002.743033:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318003.638986:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318003.766929:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318003.832357:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318003.832366:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318003.832368:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318003.832372:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318004.665440:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318004.790906:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318004.855037:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318004.855047:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318004.855050:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318004.855053:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318005.687057:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318005.815616:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318006.710961:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318006.838965:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318007.739462:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318007.862765:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318008.761627:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318008.887247:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318009.782984:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318009.910952:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318010.167913:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318010.167923:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318010.167927:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318010.167930:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318010.807216:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318010.934830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318011.832201:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318011.959300:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318012.857306:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318012.982813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318013.878927:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318014.009948:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318014.903460:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318015.030822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318015.286817:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318015.286829:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318015.286832:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318015.286835:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318015.927295:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318016.054853:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318016.950958:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318017.078831:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318017.975182:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318018.102895:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318018.998965:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318019.126823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318020.025746:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318020.151018:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318020.407062:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318020.407073:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318020.407076:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318020.407078:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318021.047096:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318021.174874:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318022.075725:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318022.200085:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318023.095427:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318023.222855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318023.289981:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317998/real 1767317998] req@ffff9d1f918d1880 x1853163904593152/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318023.290017:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1880 x1853163904593152/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318023 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318023.290037:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318023.290065:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318023.290069:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904593152:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767318023.290719:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318023.290726:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767318023.290730:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318023.290735:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3246 00000100:00080000:3.0:1767318023.290745:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318023.290751:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767318023.290788:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70380 x1853163904593408/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767318023.290817:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318023.290835:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767318023.290897:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904593408:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767318024.119004:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318024.247948:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318024.310891:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317999/real 1767317999] req@ffff9d1f918d3800 x1853163904593280/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318024 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318024.310923:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3800 x1853163904593280/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318024 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318024.310942:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318024.310965:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318024.310985:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904593280:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767318024.311073:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318024.311078:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318024.311081:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318024.311084:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767318024.311087:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318024.311091:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3247 00000100:00080000:3.0:1767318024.311101:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318024.311107:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767318024.311119:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767318024.311146:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70e00 x1853163904593536/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318024.311371:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904593536:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318025.143045:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318025.270818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318026.167845:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318026.295401:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318027.193327:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318027.319751:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318028.223706:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1767318028.344039:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318028.344050:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318028.344053:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318028.344056:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767318028.344138:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318029.238924:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1767318029.367056:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318029.367086:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318029.367090:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318029.367093:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767318029.367513:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318030.263472:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318030.391075:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318031.287783:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318031.416633:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318032.313809:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318032.440319:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318033.335041:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318033.463658:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318034.364280:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318034.487300:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318034.742949:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318034.742963:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318034.742966:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318034.742969:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318035.383621:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318035.514220:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318036.406890:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318036.534803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318037.431005:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318037.560178:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318038.454885:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318038.582798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318039.478981:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318039.606809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318039.862809:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318039.862821:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318039.862824:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318039.862827:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318040.502886:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318040.630938:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318041.528765:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318041.654807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318042.550906:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318042.683272:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318043.575066:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318043.703236:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318044.598877:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318044.727368:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318044.987361:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318044.987372:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318044.987376:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318044.987379:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318045.623711:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318045.751590:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318046.648588:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318046.775564:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318047.671794:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318047.798801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318048.695582:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318048.822826:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318023/real 1767318023] req@ffff9d1f92f70380 x1853163904593408/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318048 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318048.822860:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70380 x1853163904593408/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318048 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318048.822882:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318048.822961:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318048.822966:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904593408:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767318048.823799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318048.826288:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318048.826297:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767318048.826639:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318048.826650:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3270 00000100:00080000:1.0:1767318048.826662:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318048.826669:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767318048.826725:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0e00 x1853163904593664/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767318048.826820:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904593664:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767318048.826985:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318048.826989:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318049.719044:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318049.846812:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318024/real 1767318024] req@ffff9d1f92f70e00 x1853163904593536/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318049.846849:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70e00 x1853163904593536/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318049 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318049.846868:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318049.846886:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318049.846889:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904593536:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767318049.846982:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318049.846986:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767318049.847078:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318049.847206:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318049.847209:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767318049.847211:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318049.847216:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3271 00000100:00080000:3.0:1767318049.847225:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318049.847231:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767318049.847241:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767318049.847264:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72300 x1853163904593792/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767318049.847334:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904593792:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318050.750489:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318050.870800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318051.766962:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318051.894998:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318052.790903:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318052.919535:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318053.814913:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1767318053.881396:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318053.881406:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318053.881408:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318053.881411:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767318053.942854:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318054.839739:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1767318054.902820:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318054.902831:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318054.902835:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318054.902838:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767318054.966857:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318055.863428:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318055.991171:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318056.887091:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318057.016255:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318057.912467:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318058.038812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318058.934872:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318059.062934:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318059.959166:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318060.087820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318060.343305:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318060.343317:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318060.343320:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318060.343324:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318060.985144:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318061.113043:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318062.008072:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318062.134865:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318063.044694:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318063.161378:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318064.058702:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318064.182812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318065.078982:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318065.211144:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318065.463191:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318065.463202:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318065.463205:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318065.463208:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318066.103483:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318066.230949:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318067.129364:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318067.254797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318068.151142:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318068.279052:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318069.175071:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318069.302986:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318070.199773:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318070.326843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318070.582854:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318070.582864:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318070.582867:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318070.582887:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318071.224745:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318071.351055:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318072.246906:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318072.375041:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318073.270944:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318073.334871:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318048/real 1767318048] req@ffff9d1f918d0e00 x1853163904593664/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318073 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318073.334904:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0e00 x1853163904593664/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318073 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318073.334923:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318073.334951:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318073.334955:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904593664:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767318073.335126:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318073.335131:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767318073.335134:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318073.335139:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3296 00000100:00080000:3.0:1767318073.335148:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318073.335153:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767318073.335206:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70a80 x1853163904593920/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767318073.335229:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318073.335233:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767318073.335355:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904593920:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767318073.399660:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318074.298883:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318074.360066:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318049/real 1767318049] req@ffff9d1f92f72300 x1853163904593792/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318074.360100:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72300 x1853163904593792/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318074 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318074.360121:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318074.360151:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318074.360155:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904593792:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767318074.360911:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318074.360916:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318074.360919:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318074.360922:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767318074.360925:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318074.360931:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3297 00000100:00080000:3.0:1767318074.360941:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318074.360947:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767318074.360957:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767318074.360982:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73100 x1853163904594048/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767318074.363827:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904594048:192.168.202.140@tcp:38:kworker.0 00000100:00100000:3.0:1767318074.422812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318075.319388:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318075.446840:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318076.344295:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318076.470876:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318077.369846:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1767318077.369853:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318077.369860:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318077.369879:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318077.369883:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767318077.494805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318078.391013:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318078.519311:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318079.414937:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318079.414949:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318079.414952:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318079.414956:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318079.414970:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318079.542828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318080.440896:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318080.569293:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318081.462946:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318081.590893:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318082.486988:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318082.615222:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318083.510897:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318083.638821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318084.537438:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318084.665031:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318084.919162:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318084.919191:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318084.919193:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318084.919197:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318085.559109:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318085.686809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318086.582919:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318086.711963:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318087.608280:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318087.738202:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318088.631511:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318088.761216:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318089.658530:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318089.782895:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318090.038903:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318090.038913:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318090.038916:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318090.038920:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318090.684532:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318090.807631:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318091.703268:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318091.833385:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318092.730694:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318092.854803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318093.751520:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318093.878804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318094.774867:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318094.902804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318095.158803:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318095.158814:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318095.158817:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318095.158820:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318095.801018:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318095.926920:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318096.823319:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318096.952768:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318097.847823:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318097.974811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318098.871000:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318098.871111:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318073/real 1767318073] req@ffff9d1f92f70a80 x1853163904593920/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318098.871142:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70a80 x1853163904593920/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318098 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767318098.871163:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318098.871191:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318098.871196:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904593920:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767318098.871352:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318098.871357:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767318098.871361:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318098.871367:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3320 00000100:00080000:0.0:1767318098.871376:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318098.871383:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767318098.871416:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032680 x1853163904594176/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767318098.871449:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318098.871454:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1767318098.871598:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904594176:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767318098.998909:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318099.895247:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318099.896787:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318074/real 1767318074] req@ffff9d1f92f73100 x1853163904594048/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318099 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318099.896822:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73100 x1853163904594048/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318099 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767318099.896845:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318099.896875:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318099.896880:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904594048:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767318099.897249:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318099.897273:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318099.897275:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318099.897278:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767318099.897281:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318099.897286:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3322 00000100:00080000:0.0:1767318099.897295:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318099.897302:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767318099.897312:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767318099.897341:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031500 x1853163904594304/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767318099.902240:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904594304:192.168.202.140@tcp:38:kworker.0 00000100:00100000:3.0:1767318100.025348:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318100.918917:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318101.046806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318101.943204:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318102.073584:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318102.966961:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318103.094847:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318103.926857:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318103.926868:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318103.926871:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318103.926875:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318103.993881:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318104.119101:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318104.950825:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318104.950835:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318104.950838:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318104.950859:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318105.014901:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318105.142839:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318106.038931:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318106.166977:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318107.064600:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318107.191632:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318108.086988:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318108.214842:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318109.112814:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318109.238838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318110.009817:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318110.009826:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318110.009829:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318110.009833:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318110.134988:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318110.262806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318111.166000:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318111.287069:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318112.183906:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318112.310959:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318113.206918:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318113.334793:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318114.236091:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318114.360090:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318115.126920:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318115.126932:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318115.126936:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318115.126939:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318115.255058:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318115.383040:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318116.278943:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318116.407138:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318117.302984:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318117.431263:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318118.326995:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318118.454817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318119.351232:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318119.478943:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318120.249205:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318120.249216:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318120.249219:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318120.249222:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318120.375155:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318120.504415:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318121.400625:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318121.527390:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318122.424246:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318122.550829:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767318123.382845:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318098/real 1767318098] req@ffff9d1f91032680 x1853163904594176/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318123 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318123.382880:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032680 x1853163904594176/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318123 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767318123.382901:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318123.382931:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318123.382935:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904594176:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767318123.383036:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318123.383042:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767318123.383045:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318123.383051:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3346 00000100:00080000:0.0:1767318123.383088:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318123.383095:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767318123.383134:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030a80 x1853163904594432/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767318123.383158:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318123.383162:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1767318123.383228:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904594432:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767318123.447034:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318123.574811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767318124.407019:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318099/real 1767318099] req@ffff9d1f91031500 x1853163904594304/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318124 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318124.407054:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031500 x1853163904594304/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318124 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767318124.407076:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318124.407105:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318124.407110:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904594304:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767318124.407203:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318124.407209:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318124.407211:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318124.407215:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767318124.407218:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318124.407224:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3347 00000100:00080000:0.0:1767318124.407233:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318124.407239:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767318124.407250:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767318124.407283:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032300 x1853163904594560/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767318124.407363:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904594560:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318124.470940:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318124.598918:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318125.495191:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318125.623822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318126.519053:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318126.647184:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318127.543134:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318127.670854:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318128.438831:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318128.438844:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318128.438848:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318128.438851:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318128.567729:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318128.694803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318129.463001:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318129.463012:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318129.463015:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318129.463017:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318129.592308:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318129.719774:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318130.617020:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318130.744110:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318131.644520:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318131.766827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318132.669762:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318132.790824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318133.686885:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318133.814944:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318134.583144:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318134.583156:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318134.583159:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318134.583162:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318134.711192:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318134.838771:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318135.738034:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318135.863128:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318136.759478:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318136.886759:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318137.783404:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318137.910807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318138.806901:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318138.934899:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318139.705251:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318139.705262:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318139.705265:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318139.705269:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318139.832260:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318139.959569:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318140.858089:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318140.984498:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318141.879019:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318142.007023:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318142.903470:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318143.030793:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318143.930662:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318144.054780:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318144.823605:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318144.823616:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318144.823619:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318144.823622:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318144.950990:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318145.079069:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318145.974867:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318146.102806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318146.999857:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318147.126859:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318148.023144:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318148.150803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767318148.918898:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318123/real 1767318123] req@ffff9d1f91030a80 x1853163904594432/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318148 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318148.918934:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030a80 x1853163904594432/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318148 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318148.918977:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318148.918998:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318148.919003:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904594432:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767318148.919132:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318148.919138:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767318148.919141:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318148.919146:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3371 00000100:00080000:1.0:1767318148.919155:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318148.919162:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767318148.919190:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d2d80 x1853163904594688/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318148.919214:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318148.919218:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767318148.919281:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904594688:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767318149.046914:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318149.175229:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767318149.944918:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318124/real 1767318124] req@ffff9d1f91032300 x1853163904594560/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318149 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318149.944954:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032300 x1853163904594560/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318149 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318149.944978:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318149.945008:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318149.945013:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904594560:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767318149.945532:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318149.945538:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318149.945540:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318149.945554:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767318149.945557:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318149.945563:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3372 00000100:00080000:0.0:1767318149.945572:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318149.945579:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767318149.945591:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767318149.945625:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031180 x1853163904594816/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767318149.945780:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904594816:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318150.072099:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318150.198969:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318151.094902:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318151.222760:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318152.123428:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318152.249961:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318153.142916:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318153.273076:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318153.975295:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318153.975305:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318153.975308:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318153.975329:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318154.170275:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318154.294802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318154.998981:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318154.998991:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318154.998994:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318154.998997:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318155.192648:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318155.318793:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318156.218438:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318156.342889:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318157.239056:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318157.366815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318158.263961:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318158.391121:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318159.291814:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318159.415011:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318160.182883:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318160.182897:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318160.182901:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318160.182905:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318160.311225:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318160.438882:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318161.336197:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318161.462864:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318162.362841:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318162.487380:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318163.382916:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318163.511838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318164.413158:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318164.534803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318165.305828:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318165.305838:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318165.305841:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318165.305844:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318165.434438:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318165.558824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318166.459801:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318166.582946:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318167.479308:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318167.606861:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318168.503569:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318168.632064:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318169.529592:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318169.654855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318170.422938:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318170.422952:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318170.422956:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318170.422959:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318170.554336:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318170.679175:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318171.575167:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318171.702798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318172.598877:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318172.726825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318173.430828:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318148/real 1767318148] req@ffff9d1f918d2d80 x1853163904594688/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318173 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318173.430866:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d2d80 x1853163904594688/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318173 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318173.430888:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318173.430918:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318173.430923:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904594688:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767318173.432717:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318173.432728:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767318173.432732:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318173.432742:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3396 00000100:00080000:1.0:1767318173.432772:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318173.432779:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767318173.432813:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0000 x1853163904594944/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318173.432845:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318173.432849:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767318173.432935:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904594944:192.168.202.140@tcp:250:kworker.0 00010000:00010000:0.0:1767318173.622849:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318173.752085:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318174.454861:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318149/real 1767318149] req@ffff9d1f91031180 x1853163904594816/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318174.454902:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031180 x1853163904594816/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318174 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318174.454925:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318174.454955:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318174.454960:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904594816:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767318174.455220:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318174.455225:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318174.455228:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318174.455232:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767318174.455235:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318174.455240:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3397 00000100:00080000:1.0:1767318174.455250:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318174.455257:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767318174.455301:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767318174.455333:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1f80 x1853163904595072/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318174.455482:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904595072:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318174.648060:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318174.775406:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318175.671078:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318175.800176:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318176.695001:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318176.822837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318177.718923:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318177.846812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318178.487089:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318178.487100:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318178.487103:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318178.487144:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318178.745136:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318178.874618:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318179.511066:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318179.511077:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318179.511080:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318179.511084:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318179.770525:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318179.897768:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318180.792039:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318180.918799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318181.815427:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318181.943361:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318182.838893:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318182.967278:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318183.863274:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318183.990800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318184.761597:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318184.761618:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318184.761621:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318184.761626:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318184.887052:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318185.014970:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318185.910947:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318186.042070:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318186.934977:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318187.063327:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318187.962009:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318188.087182:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318188.983379:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318189.112227:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318189.879261:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318189.879284:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318189.879288:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318189.879292:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318190.007263:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318190.134841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318191.032511:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318191.159767:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318192.054896:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318192.182800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318193.079106:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318193.208244:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318194.103843:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318194.232869:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318194.999389:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318194.999417:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318194.999420:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318194.999423:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318195.127009:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318195.254974:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318196.153117:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318196.278811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318197.176107:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318197.302858:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318198.199218:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318198.326798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318198.966856:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318173/real 1767318173] req@ffff9d1f918d0000 x1853163904594944/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318198.966900:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0000 x1853163904594944/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318198 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318198.966919:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318198.966945:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318198.966949:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904594944:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767318198.966980:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318198.966998:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767318198.967001:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767318198.967005:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3421 00000100:00080000:2.0:1767318198.967012:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767318198.967017:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767318198.967073:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cd880 x1853163904595200/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767318198.967107:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318198.967112:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767318198.967175:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904595200:192.168.202.140@tcp:250:kworker.0 00010000:00010000:1.0:1767318199.222899:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318199.351437:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318199.991415:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318174/real 1767318174] req@ffff9d1f918d1f80 x1853163904595072/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318199.991459:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1f80 x1853163904595072/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318199 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318199.991479:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318199.991505:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318199.991510:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904595072:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767318199.991534:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318199.991540:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318199.991543:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318199.991546:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767318199.991549:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318199.991554:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3422 00000100:00080000:3.0:1767318199.991562:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318199.991569:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767318199.991579:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767318199.991609:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73100 x1853163904595328/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318199.991726:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904595328:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767318200.246921:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318200.374778:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318201.274314:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318201.398805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318202.294940:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318202.422863:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318203.319115:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318203.448228:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318204.025955:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318204.025977:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318204.025981:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318204.025983:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318204.342858:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318204.470812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318205.047147:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318205.047158:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318205.047161:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318205.047164:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318205.369820:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318205.497336:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318206.392454:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318206.519103:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318207.418839:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318207.547512:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318208.440257:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318208.567097:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318209.465594:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318209.594733:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318210.358864:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318210.358879:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318210.358894:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318210.358899:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318210.487233:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318210.616148:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318211.511001:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318211.639595:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318212.538883:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318212.662988:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318213.559762:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318213.686864:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318214.586024:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318214.711810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318215.480576:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318215.480593:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318215.480597:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318215.480600:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318215.608546:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318215.735023:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318216.633223:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318216.758969:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318217.655718:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318217.782846:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318218.678999:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318218.806796:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318219.706958:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318219.830810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318220.599069:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318220.599079:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318220.599082:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318220.599086:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318220.728624:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318220.854804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318221.750945:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318221.879312:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318222.775257:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318222.902835:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318223.478845:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318198/real 1767318198] req@ffff9d1f967cd880 x1853163904595200/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318223 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318223.478878:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cd880 x1853163904595200/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318223 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318223.478897:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318223.478926:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318223.478930:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904595200:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767318223.479277:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318223.479283:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767318223.479286:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318223.479291:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3446 00000100:00080000:3.0:1767318223.479299:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318223.479305:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767318223.479337:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70a80 x1853163904595456/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767318223.479362:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318223.479389:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767318223.479497:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904595456:192.168.202.140@tcp:250:kworker.0 00010000:00010000:3.0:1767318223.798964:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318223.926800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318224.502827:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318199/real 1767318199] req@ffff9d1f92f73100 x1853163904595328/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318224 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318224.502864:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73100 x1853163904595328/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318224 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318224.502887:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318224.502916:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318224.502920:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904595328:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767318224.503014:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318224.503020:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318224.503023:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318224.503026:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767318224.503029:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318224.503035:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3447 00000100:00080000:3.0:1767318224.503045:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318224.503052:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767318224.503064:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767318224.503098:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71c00 x1853163904595584/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318224.503185:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904595584:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318224.822982:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318224.950809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318225.846898:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318225.974959:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318226.870873:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318226.998940:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318227.895467:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318228.023056:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318228.536191:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318228.536202:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318228.536223:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318228.536226:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318228.919280:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318229.046883:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318229.559157:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318229.559166:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318229.559169:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318229.559191:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318229.943099:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318230.070883:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318230.968917:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318231.094790:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318231.991636:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318232.119374:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318233.016464:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318233.143151:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318234.045535:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318234.167369:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318234.936494:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318234.936504:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318234.936507:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318234.936510:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318235.063801:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318235.190827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318236.086928:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318236.214771:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318237.111216:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318237.238838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318238.135250:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318238.263271:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318239.158836:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318239.287582:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318240.057126:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318240.057136:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318240.057139:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318240.057142:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318240.182880:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318240.310797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318241.206947:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318241.335070:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318242.230855:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318242.358916:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318243.255184:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318243.383128:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318244.284407:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318244.408971:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318245.176740:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318245.176749:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318245.176752:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318245.176755:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318245.303103:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318245.430812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318246.327929:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318246.456923:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318247.353315:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318247.478795:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318248.374986:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318248.502815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767318249.015871:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318224/real 1767318224] req@ffff9d1f92f71c00 x1853163904595584/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318249 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318249.015919:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71c00 x1853163904595584/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318249 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318249.015932:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318223/real 1767318223] req@ffff9d1f92f70a80 x1853163904595456/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318249.015944:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70a80 x1853163904595456/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318248 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318249.015973:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318249.016002:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318249.016007:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904595584:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767318249.016013:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318249.016019:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318249.016022:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904595456:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767318249.016062:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318249.016069:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767318249.016072:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318249.016078:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3471 00000100:00080000:0.0:1767318249.016158:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318249.016164:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767318249.016202:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031180 x1853163904595712/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767318249.016227:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318249.016231:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767318249.016248:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318249.016252:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3472 00000100:00080000:0.0:1767318249.016256:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318249.016260:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767318249.016268:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904595712:192.168.202.140@tcp:250:kworker.0 00000008:00100000:0.0:1767318249.016270:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767318249.016281:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030000 x1853163904595840/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767318249.016429:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904595840:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318249.399840:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318249.526760:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318250.422943:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318250.551130:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318251.446917:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318251.574822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318252.470890:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318252.598805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318253.495159:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318253.622823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318254.391180:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318254.391201:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318254.391204:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318254.391208:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318254.519079:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318254.646820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318255.543007:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318255.670803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318256.566916:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318256.694817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318257.590947:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318257.718813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318258.614915:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318258.743338:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318259.511288:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318259.511295:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318259.511298:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318259.511300:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318259.638929:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318259.766801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318260.663078:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318260.790801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318261.687109:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318261.816176:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318262.710919:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318262.839088:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318263.734916:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318263.863504:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318264.630847:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318264.630858:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318264.630861:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318264.630865:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318264.759874:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318264.886799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318265.783988:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318265.913553:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318266.806874:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318266.934812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318267.838035:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318267.958809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318268.857984:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318268.982832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318269.751217:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318269.751229:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318269.751232:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318269.751235:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318269.879177:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318270.006763:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318270.903294:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318271.031334:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318271.933781:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318272.056133:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318272.952393:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318273.080170:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318273.977201:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318274.104460:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318274.870913:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318274.870923:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318274.870926:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318274.870929:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767318274.871175:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318249/real 1767318249] req@ffff9d1f91030000 x1853163904595840/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318274 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318274.871204:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030000 x1853163904595840/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318274 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318274.871214:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318249/real 1767318249] req@ffff9d1f91031180 x1853163904595712/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318274 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318274.871224:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031180 x1853163904595712/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318274 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318274.871241:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318274.871261:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318274.871266:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904595840:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767318274.871271:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318274.871276:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318274.871279:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904595712:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767318274.871693:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318274.871698:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767318274.871701:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318274.871706:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3496 00000100:00080000:3.0:1767318274.871715:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318274.871721:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767318274.871751:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70a80 x1853163904595968/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767318274.871779:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318274.871783:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767318274.871785:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318274.871789:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3496 00000100:00080000:3.0:1767318274.871792:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318274.871796:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767318274.871805:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767318274.871818:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71c00 x1853163904596096/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767318274.871885:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904595968:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767318274.871979:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904596096:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318275.010468:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318275.127703:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318276.022881:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318276.152387:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318277.046939:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318277.174800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318278.070906:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318278.198815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318279.095009:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318279.222865:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318279.994429:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318279.994439:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318279.994443:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318279.994462:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318280.122254:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318280.246879:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318281.143532:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318281.271821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318282.166942:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318282.295040:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318283.190992:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318283.320414:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318284.215387:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318284.342835:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318285.110881:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318285.110893:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318285.110896:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318285.110900:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318285.239147:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318285.368058:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318286.263909:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318286.390978:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318287.287351:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318287.414807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318288.311022:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318288.438810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318289.334898:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318289.462801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318290.230870:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318290.230881:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318290.230884:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318290.230888:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318290.358967:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318290.486807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318291.384974:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318291.510996:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318292.407144:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318292.534804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318293.432468:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318293.558802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318294.455011:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318294.582818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318295.361183:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318295.361194:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318295.361198:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318295.361201:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318295.479190:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318295.606810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318296.502913:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318296.630820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318297.527189:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318297.654952:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318298.550905:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318298.678794:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767318299.446812:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318274/real 1767318274] req@ffff9d1f92f70a80 x1853163904595968/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318299 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318299.446848:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70a80 x1853163904595968/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318299 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318299.446859:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318274/real 1767318274] req@ffff9d1f92f71c00 x1853163904596096/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318299 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318299.446871:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71c00 x1853163904596096/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318299 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318299.446896:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318299.446921:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318299.446925:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904595968:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767318299.446931:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318299.446937:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318299.446940:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904596096:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767318299.447158:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318299.447164:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767318299.447168:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318299.447173:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3522 00000100:00080000:3.0:1767318299.447182:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318299.447190:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767318299.447221:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73100 x1853163904596224/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767318299.447246:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318299.447250:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767318299.447251:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318299.447255:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3522 00000100:00080000:3.0:1767318299.447259:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318299.447262:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767318299.447272:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767318299.447283:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72300 x1853163904596352/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767318299.447356:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904596224:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767318299.447439:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904596352:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318299.578265:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318299.702942:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318300.598905:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318300.726879:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318301.625895:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318301.750875:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318302.648650:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318302.775015:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318303.671596:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318303.798802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318304.567170:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318304.567181:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318304.567184:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318304.567187:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318304.694846:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318304.822797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318305.719489:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318305.846902:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318306.743205:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318306.871254:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318307.770209:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318307.897880:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318308.791074:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318308.918805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318309.687056:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318309.687066:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318309.687069:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318309.687072:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318309.814937:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318309.943313:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318310.838936:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318310.966877:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318311.862866:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318311.991117:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318312.887132:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318313.014955:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318313.911646:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318314.038802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318314.806871:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318314.806881:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318314.806901:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318314.806905:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318314.935718:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318315.066038:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318315.958943:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318316.086864:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318316.984306:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318317.110821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318318.007627:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318318.134809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318319.032508:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318319.158795:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318319.928712:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318319.928723:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318319.928726:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318319.928729:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318320.054987:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318320.182791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318321.079000:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318321.206861:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318322.103039:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318322.231770:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318323.130004:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318323.254802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767318324.023611:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318299/real 1767318299] req@ffff9d1f92f73100 x1853163904596224/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318324 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318324.023646:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73100 x1853163904596224/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318324 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318324.023711:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318299/real 1767318299] req@ffff9d1f92f72300 x1853163904596352/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318324 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318324.023727:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72300 x1853163904596352/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318324 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318324.023744:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318324.023772:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318324.023777:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904596224:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767318324.023782:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318324.023788:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318324.023791:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904596352:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767318324.026122:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318324.026141:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767318324.026145:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318324.026153:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3547 00000100:00080000:0.0:1767318324.026163:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318324.026170:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767318324.026205:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032d80 x1853163904596480/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767318324.026238:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318324.026242:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767318324.026244:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318324.026249:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3547 00000100:00080000:0.0:1767318324.026253:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318324.026257:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767318324.026266:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767318324.026279:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031880 x1853163904596608/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767318324.026412:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904596480:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767318324.026516:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904596608:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318324.152360:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318324.278825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318325.178166:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318325.303323:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318326.199125:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318326.326800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318327.223785:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318327.352732:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318328.246942:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318328.374793:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318329.143212:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318329.143223:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318329.143226:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318329.143229:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318329.277226:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318329.402757:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318330.298443:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318330.422855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318331.321270:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318331.446807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318332.343503:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318332.471236:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318333.366872:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318333.495176:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318334.262920:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318334.262930:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318334.262933:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318334.262937:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318334.390965:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318334.518823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318335.414889:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318335.542829:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318336.438936:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318336.566892:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318337.463019:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318337.590904:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318338.495813:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318338.614813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318339.384398:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318339.384408:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318339.384411:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318339.384415:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318339.512398:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318339.638794:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318340.536021:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318340.662813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318341.566041:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318341.687870:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318342.582838:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318342.710883:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318343.607577:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318343.734826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318344.503045:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318344.503057:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318344.503060:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318344.503064:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318344.632202:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318344.758807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318345.655089:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318345.782771:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318346.690218:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318346.807946:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318347.702957:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318347.830845:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318348.727923:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318348.854857:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318349.622807:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318349.622816:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318349.622819:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1767318349.622821:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318324/real 1767318324] req@ffff9d1f91032d80 x1853163904596480/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318349 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767318349.622822:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767318349.622860:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032d80 x1853163904596480/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318349 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318349.622870:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318324/real 1767318324] req@ffff9d1f91031880 x1853163904596608/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318349 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318349.622881:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031880 x1853163904596608/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318349 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318349.622899:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318349.622929:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318349.622934:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904596480:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767318349.622941:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318349.622947:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318349.622950:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904596608:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767318349.623132:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318349.623136:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767318349.623153:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318349.623157:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3571 00000100:00080000:1.0:1767318349.623165:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318349.623216:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767318349.623251:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1f80 x1853163904596736/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318349.623279:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318349.623283:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767318349.623285:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318349.623289:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3571 00000100:00080000:1.0:1767318349.623293:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318349.623296:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767318349.623305:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767318349.623314:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0000 x1853163904596864/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318349.623415:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904596736:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767318349.623500:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904596864:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318349.752074:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318349.878883:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318350.774851:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318350.905927:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318351.798961:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318351.929000:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318352.823984:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318352.950812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318353.847092:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318353.976394:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318354.742870:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318354.742880:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318354.742883:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318354.742886:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1767318354.871428:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318355.001088:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318355.896285:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318356.024152:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318356.918913:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318357.047120:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318357.942924:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318358.070809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318358.968031:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318359.094824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318359.868878:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318359.868888:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318359.868891:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318359.868894:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318359.990913:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318360.120990:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318361.014906:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318361.142772:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318362.039274:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318362.166801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318363.067600:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318363.190870:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318364.086989:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318364.214830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318364.982792:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318364.982802:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318364.982805:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318364.982808:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318365.115175:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318365.238807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318366.134982:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318366.263692:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318367.159380:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318367.287339:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318368.184267:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318368.310893:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318369.215075:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318369.336037:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318370.102822:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318370.102832:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318370.102836:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318370.102840:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318370.231737:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318370.358811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318371.254820:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318371.382815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318372.279916:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318372.407500:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318373.303930:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318373.430797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767318374.199872:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318349/real 1767318349] req@ffff9d1f918d1f80 x1853163904596736/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318374.199906:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1f80 x1853163904596736/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318374 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318374.199915:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318349/real 1767318349] req@ffff9d1f918d0000 x1853163904596864/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318374.199925:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0000 x1853163904596864/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318374 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767318374.199943:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318374.199966:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318374.199970:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904596736:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767318374.199994:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318374.200001:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318374.200004:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904596864:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767318374.200117:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318374.200122:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767318374.200125:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767318374.200131:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3597 00000100:00080000:2.0:1767318374.200141:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767318374.200148:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767318374.200177:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cfb80 x1853163904596992/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767318374.200200:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318374.200203:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767318374.200205:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767318374.200208:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3597 00000100:00080000:2.0:1767318374.200211:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767318374.200215:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767318374.200223:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767318374.200235:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cd180 x1853163904597120/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767318374.200309:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904596992:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767318374.200411:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904597120:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318374.330240:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318374.454796:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318375.351306:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318375.480544:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318376.375621:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318376.503601:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318377.399658:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318377.528209:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318378.423137:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318378.550814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318379.321162:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318379.321173:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318379.321176:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318379.321179:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1767318379.446850:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318379.576067:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318380.470916:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318380.599196:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318381.494983:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318381.624110:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318382.518999:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318382.646799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318383.542860:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318383.671297:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318384.442358:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318384.442369:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318384.442391:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318384.442394:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318384.566881:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318384.694810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318385.595564:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318385.718808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318386.620453:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318386.742793:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318387.638839:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318387.766795:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318388.664096:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318388.790798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318389.558817:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318389.558828:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318389.558849:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318389.558853:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318389.687725:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318389.817737:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318390.711095:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318390.838799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318391.735826:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318391.863761:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318392.760402:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318392.886812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318393.783128:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318393.910920:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318394.680290:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318394.680301:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318394.680305:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318394.680308:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318394.807207:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318394.934864:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318395.830815:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318395.959014:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318396.855149:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318396.982807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318397.879488:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318398.006791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318398.902911:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318399.030813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318399.798789:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318399.798799:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318399.798803:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318399.798806:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767318399.798856:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318374/real 1767318374] req@ffff9d1f967cfb80 x1853163904596992/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318399 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318399.798889:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cfb80 x1853163904596992/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318399 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318399.798901:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318374/real 1767318374] req@ffff9d1f967cd180 x1853163904597120/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318399 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318399.798912:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cd180 x1853163904597120/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318399 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318399.798947:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318399.798973:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318399.798977:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904596992:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767318399.798983:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318399.798987:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318399.798990:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904597120:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767318399.799110:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318399.799115:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767318399.799119:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767318399.799132:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3621 00000100:00080000:2.0:1767318399.799141:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767318399.799147:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767318399.799176:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cc380 x1853163904597248/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767318399.799204:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318399.799236:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767318399.799238:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767318399.799243:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3621 00000100:00080000:2.0:1767318399.799247:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767318399.799251:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767318399.799259:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767318399.799272:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cc000 x1853163904597376/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318399.799710:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904597248:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767318399.799800:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904597376:192.168.202.140@tcp:38:kworker.0 00010000:00010000:2.0:1767318399.928342:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318400.055118:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318400.950972:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318401.078807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318401.974899:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318402.105326:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318403.000991:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318403.127333:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318404.022914:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318404.150892:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318404.918894:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318404.918909:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318404.918912:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318404.918916:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318405.047003:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318405.174968:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318406.071108:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318406.201116:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318407.096979:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318407.222866:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318408.118920:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318408.246865:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318409.142987:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318409.271438:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318410.038919:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318410.038935:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318410.038949:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318410.038952:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318410.170568:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318410.295054:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318411.191003:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318411.319755:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:2.0:1767318412.216754:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318412.342824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318413.239149:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318413.366798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318414.263402:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318414.390809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318415.158898:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318415.158913:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318415.158917:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318415.158920:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318415.287733:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318415.414799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318416.311941:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318416.442673:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318417.336877:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318417.462790:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318418.360511:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318418.490842:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318419.382902:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318419.510834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318420.280717:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318420.280728:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318420.280731:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318420.280734:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318420.406854:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318420.537225:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318421.431386:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318421.565096:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318422.455151:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318422.582808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318423.479331:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318423.606886:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318424.374932:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318399/real 1767318399] req@ffff9d1f967cc380 x1853163904597248/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318424 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318424.374968:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cc380 x1853163904597248/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318424 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318424.374978:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318399/real 1767318399] req@ffff9d1f967cc000 x1853163904597376/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318424 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318424.374999:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cc000 x1853163904597376/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318424 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318424.375048:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318424.375071:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318424.375076:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904597248:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767318424.375082:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318424.375087:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318424.375090:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904597376:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767318424.375634:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318424.375640:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767318424.375643:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318424.375649:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3647 00000100:00080000:3.0:1767318424.375658:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318424.375664:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767318424.375708:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70e00 x1853163904597504/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767318424.375724:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318424.375733:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767318424.375735:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318424.375750:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3647 00000100:00080000:3.0:1767318424.375755:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318424.375759:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767318424.375767:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767318424.375779:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70380 x1853163904597632/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318424.376529:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904597504:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767318424.376636:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904597632:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318424.503036:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318424.630859:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318425.527155:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318425.654807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318426.551406:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318426.678802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318427.574911:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318427.702827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318428.599177:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318428.726821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318429.495397:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318429.495409:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318429.495412:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318429.495416:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318429.630159:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318429.750832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318430.647654:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318430.774797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318431.671233:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318431.798801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318432.694985:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318432.822952:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318433.718985:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318433.846922:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318434.615614:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318434.615624:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318434.615635:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318434.615639:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318434.742913:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318434.870783:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318435.766917:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318435.894810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318436.791161:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318436.918794:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318437.815036:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318437.943219:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318438.839712:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318438.966820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318439.734821:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318439.734831:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318439.734834:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318439.734837:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318439.862891:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318439.990976:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318440.893949:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318441.015117:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318441.911285:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318442.038824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318442.934930:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318443.062803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318443.959467:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318444.086889:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318444.854855:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318444.854867:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318444.854870:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318444.854873:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318444.983333:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318445.110785:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318446.008214:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318446.135671:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318447.032581:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318447.158814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318448.055000:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318448.182901:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318449.078977:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318449.207291:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767318449.974826:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318424/real 1767318424] req@ffff9d1f92f70e00 x1853163904597504/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318449.974867:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70e00 x1853163904597504/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318449 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318449.974878:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318424/real 1767318424] req@ffff9d1f92f70380 x1853163904597632/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318449.974890:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70380 x1853163904597632/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318449 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318449.974907:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318449.974952:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318449.974956:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904597504:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767318449.974963:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318449.974968:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318449.974971:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904597632:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767318449.975057:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318449.975063:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767318449.975066:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318449.975072:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3672 00000100:00080000:1.0:1767318449.975081:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318449.975087:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767318449.975118:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d2d80 x1853163904597760/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318449.975144:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318449.975147:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767318449.975149:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318449.975152:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3672 00000100:00080000:1.0:1767318449.975155:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318449.975159:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767318449.975167:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767318449.975179:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0e00 x1853163904597888/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767318449.975249:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904597760:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767318449.975345:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904597888:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767318450.103521:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318450.230862:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318451.126934:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318451.254968:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318452.150957:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318452.281125:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318453.174951:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318453.303297:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318454.199802:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318454.326816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318455.096107:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318455.096119:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318455.096122:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318455.096125:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318455.225951:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318455.350819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318456.247335:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318456.374815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318457.271049:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318457.399731:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318458.294905:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318458.422853:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318459.320766:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318459.446751:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318460.214966:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318460.214977:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318460.214980:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318460.214983:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318460.342948:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318460.470870:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318461.368274:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318461.494807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318462.393106:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318462.518762:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318463.414915:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318463.544092:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318464.439830:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318464.566805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318465.335865:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318465.335877:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318465.335880:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318465.335883:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318465.463907:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318465.593456:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318466.486925:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318466.615672:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318467.511344:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318467.638807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318468.535960:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318468.662808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318469.561023:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318469.686834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318470.454881:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318470.454893:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318470.454896:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318470.454900:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318470.582984:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318470.710974:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318471.606893:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318471.734817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318472.631177:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318472.758791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318473.655237:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318473.782799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318474.551090:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318449/real 1767318449] req@ffff9d1f918d2d80 x1853163904597760/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318474 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318474.551145:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d2d80 x1853163904597760/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318474 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318474.551157:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318449/real 1767318449] req@ffff9d1f918d0e00 x1853163904597888/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318474 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318474.551168:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0e00 x1853163904597888/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318474 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318474.551212:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318474.551242:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318474.551247:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904597760:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767318474.551253:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318474.551258:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318474.551262:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904597888:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767318474.551609:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318474.551616:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767318474.551619:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318474.551625:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3697 00000100:00080000:1.0:1767318474.551640:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318474.551647:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767318474.551708:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3800 x1853163904598016/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318474.551739:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318474.551743:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767318474.551745:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318474.551749:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3697 00000100:00080000:1.0:1767318474.551752:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318474.551756:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767318474.551767:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767318474.551779:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1880 x1853163904598144/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318474.552577:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904598016:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767318474.552712:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904598144:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767318474.679343:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318474.807723:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318475.703809:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318475.835377:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318476.727041:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318476.855142:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318477.751026:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318477.879345:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318478.776377:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318478.903719:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318479.678316:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318479.678325:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318479.678328:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318479.678331:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318479.798897:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318479.926757:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318480.822964:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318480.951260:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318481.849358:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318481.974928:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318482.870928:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318482.999297:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318483.895002:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318484.023804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318484.793576:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318484.793591:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318484.793594:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318484.793598:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318484.919032:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318485.046867:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318485.943029:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318486.072304:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318486.971141:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318487.095165:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318487.992408:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318488.120730:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318489.015889:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318489.143513:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318489.919483:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318489.919497:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318489.919500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318489.919504:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318490.039306:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318490.167237:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318491.064896:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318491.190793:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318492.087156:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318492.215228:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318493.114107:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318493.239276:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318494.137820:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318494.262852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318495.035026:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318495.035038:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318495.035041:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318495.035045:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318495.160015:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318495.286871:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318496.185896:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318496.310933:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318497.206968:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318497.334886:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318498.236720:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318498.417590:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767318499.126967:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318474/real 1767318474] req@ffff9d1f918d3800 x1853163904598016/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318499 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318499.127022:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3800 x1853163904598016/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318499 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318499.127033:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318474/real 1767318474] req@ffff9d1f918d1880 x1853163904598144/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318499 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318499.127045:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1880 x1853163904598144/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318499 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318499.127069:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318499.127100:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318499.127104:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904598016:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767318499.127111:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318499.127116:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318499.127119:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904598144:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767318499.128223:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318499.128233:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767318499.128237:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318499.128247:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3722 00000100:00080000:1.0:1767318499.128260:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318499.128268:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767318499.128336:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1c00 x1853163904598272/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318499.128372:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318499.128399:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767318499.128401:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318499.128406:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3722 00000100:00080000:1.0:1767318499.128411:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318499.128416:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767318499.128427:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767318499.128440:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0700 x1853163904598400/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767318499.128731:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904598272:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767318499.128865:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904598400:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767318499.255590:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318499.447261:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318500.279104:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318500.472133:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318501.304357:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318501.495839:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318502.328741:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318502.520634:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318503.354259:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318503.543038:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318504.247097:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318504.247110:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318504.247113:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318504.247117:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318504.380544:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318504.567987:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318505.423377:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318505.591301:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318506.426515:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318506.618231:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318507.451096:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318507.638940:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318508.472001:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318508.662974:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318509.370028:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318509.370048:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318509.370052:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318509.370056:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318509.495504:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318509.687493:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318510.519541:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318510.710975:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318511.543068:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318511.735211:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318512.567389:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318512.761645:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318513.591096:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318513.785366:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318514.497217:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318514.497226:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318514.497230:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318514.497233:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318514.615503:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318514.809025:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318515.639622:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318515.830870:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318516.663101:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318516.856502:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318517.687215:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318517.878855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318518.712905:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318518.902830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318519.607077:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318519.607093:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318519.607096:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318519.607099:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318519.734926:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318519.927727:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318520.759361:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318520.950851:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318521.783166:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318521.974854:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318522.807204:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318522.998828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318523.832280:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318524.022807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318524.726832:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318499/real 1767318499] req@ffff9d1f918d1c00 x1853163904598272/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318524.726865:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1c00 x1853163904598272/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318524 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318524.726877:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318499/real 1767318499] req@ffff9d1f918d0700 x1853163904598400/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318524.726888:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0700 x1853163904598400/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318524 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318524.726941:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318524.726951:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318524.726955:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904598272:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767318524.726961:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318524.726966:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318524.726969:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904598400:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767318524.727588:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318524.727600:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767318524.727603:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318524.727608:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3746 00000100:00080000:1.0:1767318524.727617:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318524.727622:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767318524.727650:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3480 x1853163904598528/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318524.727700:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318524.727704:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767318524.727706:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318524.727710:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3746 00000100:00080000:1.0:1767318524.727713:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318524.727717:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767318524.727725:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767318524.727735:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1180 x1853163904598656/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318524.727882:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904598528:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767318524.727995:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904598656:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318524.855149:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318525.047036:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318525.879255:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318526.076787:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318526.903029:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318527.096498:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318527.926918:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318528.119204:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318528.950917:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318529.142809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318529.847380:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318529.847391:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318529.847394:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318529.847397:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318529.974974:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318530.166855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318530.999243:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318531.191004:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318532.024404:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318532.214823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318533.046935:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318533.239259:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318534.072608:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318534.262978:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318534.966810:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318534.966821:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318534.966825:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318534.966829:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318535.095244:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318535.287103:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318536.119869:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318536.310848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318537.143018:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318537.334784:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318538.169155:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318538.358809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318539.192361:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318539.382826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318540.087510:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318540.087521:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318540.087524:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318540.087528:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318540.217285:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318540.406791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318541.239528:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318541.430808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318542.263106:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318542.455715:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318543.287105:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318543.478778:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318544.310898:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318544.505577:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318545.207205:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318545.207215:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318545.207218:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318545.207221:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318545.336820:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318545.529368:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318546.359580:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318546.551225:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318547.383079:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318547.574951:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318548.406927:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318548.599293:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767318549.302971:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318524/real 1767318524] req@ffff9d1f918d3480 x1853163904598528/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318549 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318549.303005:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3480 x1853163904598528/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318549 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318549.303016:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318524/real 1767318524] req@ffff9d1f918d1180 x1853163904598656/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318549 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318549.303027:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1180 x1853163904598656/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318549 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318549.303046:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318549.303070:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318549.303075:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904598528:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767318549.303081:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318549.303097:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318549.303101:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904598656:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767318549.303184:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318549.303190:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767318549.303193:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318549.303198:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3772 00000100:00080000:3.0:1767318549.303207:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318549.303213:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767318549.303246:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70700 x1853163904598784/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767318549.303269:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318549.303273:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767318549.303284:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318549.303288:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3772 00000100:00080000:3.0:1767318549.303291:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318549.303295:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767318549.303303:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767318549.303315:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73800 x1853163904598912/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767318549.303377:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904598784:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767318549.303479:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904598912:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318549.431931:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318549.622847:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318550.455236:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318550.646813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318551.479264:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318551.672000:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318552.506024:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318552.694843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318553.533375:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318553.719630:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318554.423612:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318554.423638:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318554.423641:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318554.423645:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318554.551900:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318554.742798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318555.575112:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318555.767354:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318556.603153:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318556.790912:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318557.623199:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318557.814807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318558.647036:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318558.838832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318559.544257:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318559.544266:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318559.544270:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318559.544273:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318559.670943:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318559.862805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318560.695488:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318560.887155:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318561.721596:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318561.910821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318562.742956:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318562.936512:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318563.770042:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318563.958850:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318564.663284:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318564.663294:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318564.663297:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318564.663301:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318564.791565:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318564.982991:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318565.818593:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318566.006808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318566.855745:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318567.031037:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318567.862969:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318568.055012:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318568.888147:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318569.083859:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318569.783308:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318569.783323:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318569.783327:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318569.783330:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318569.912070:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318570.103017:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318570.935121:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318571.127035:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318571.958938:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318572.150806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318572.983011:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318573.174865:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318574.006914:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318574.198827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318574.903385:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318549/real 1767318549] req@ffff9d1f92f70700 x1853163904598784/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318574.903402:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318574.903409:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318574.903412:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318574.903416:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767318574.903420:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70700 x1853163904598784/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318574 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318574.903432:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318549/real 1767318549] req@ffff9d1f92f73800 x1853163904598912/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318574.903444:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73800 x1853163904598912/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318574 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318574.903463:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318574.903484:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318574.903489:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904598784:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767318574.903527:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318574.903536:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00080000:2.0:1767318574.903539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1767318574.903542:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904598912:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767318574.903545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767318574.903551:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767318574.903557:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3796 00000100:00080000:2.0:1767318574.903567:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767318574.903584:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767318574.903615:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cce00 x1853163904599040/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767318574.903654:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318574.903658:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767318574.903660:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767318574.903663:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3796 00000100:00080000:2.0:1767318574.903666:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767318574.903670:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767318574.903886:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904599040:192.168.202.140@tcp:250:kworker.0 00000008:00100000:2.0:1767318574.904825:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767318574.904849:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cc700 x1853163904599168/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318574.908534:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904599168:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318575.031033:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318575.222981:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318576.054961:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318576.247744:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318577.082421:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318577.270796:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318578.105178:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318578.299284:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318579.127481:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318579.318824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318580.023012:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318580.023024:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318580.023028:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318580.023031:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318580.151399:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318580.343761:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318581.178594:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318581.366911:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318582.198970:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318582.390873:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318583.227503:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318583.415131:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318584.246941:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318584.438813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318585.142863:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318585.142876:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318585.142880:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318585.142884:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318585.270975:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318585.462813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318586.295746:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318586.487425:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318587.319184:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318587.511031:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318588.343075:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318588.536376:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318589.372113:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318589.558803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318590.264531:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318590.264543:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318590.264546:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318590.264549:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318590.391009:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318590.582875:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318591.415331:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318591.606811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318592.440720:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318592.630827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318593.462944:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318593.654825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318594.487165:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318594.679097:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318595.383083:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318595.383096:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318595.383099:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318595.383101:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318595.514025:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318595.702815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318596.535500:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318596.726838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318597.561386:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318597.752429:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318598.584494:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318598.774832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767318599.479535:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318574/real 1767318574] req@ffff9d1f967cc700 x1853163904599168/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318599.479578:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cc700 x1853163904599168/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318599 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318599.479590:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318574/real 1767318574] req@ffff9d1f967cce00 x1853163904599040/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318599.479600:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cce00 x1853163904599040/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318599 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318599.479623:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318599.479643:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318599.479669:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904599168:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767318599.479675:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318599.479718:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318599.479721:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904599040:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767318599.479878:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318599.479884:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767318599.479888:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318599.479895:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3822 00000100:00080000:1.0:1767318599.479907:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318599.479914:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767318599.479951:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1500 x1853163904599296/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318599.479977:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318599.479980:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767318599.479982:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318599.479985:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3822 00000100:00080000:1.0:1767318599.479989:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318599.479992:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767318599.480022:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767318599.480035:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d2300 x1853163904599424/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767318599.480890:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904599296:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767318599.481015:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904599424:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318599.607095:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318599.798847:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318600.631406:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318600.822858:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318601.661739:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318601.846815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318602.678929:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318602.870886:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318603.716186:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318603.894848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318604.598922:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318604.598930:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318604.598932:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318604.598935:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318604.738024:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318604.918809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318605.751032:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318605.943130:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318606.778140:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318606.966939:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318607.800305:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318607.991625:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318608.824579:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318609.014889:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318609.718807:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318609.718817:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318609.718820:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318609.718824:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318609.846898:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318610.039266:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318610.872206:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318611.062830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318611.898128:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318612.086916:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318612.920903:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318613.111022:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318613.942905:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318614.138969:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318614.840324:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318614.840334:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318614.840337:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318614.840340:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318614.968498:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318615.158821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318615.990940:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318616.182855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318617.014933:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318617.206843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318618.040181:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318618.232629:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318619.063271:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318619.255721:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318619.960324:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318619.960334:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318619.960338:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318619.960341:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318620.087518:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318620.278800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318621.110922:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318621.307790:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318622.134902:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318622.330058:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318623.159446:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318623.352357:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767318624.054793:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318599/real 1767318599] req@ffff9d1f918d1500 x1853163904599296/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318624.054831:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1500 x1853163904599296/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318624 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318624.054858:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318599/real 1767318599] req@ffff9d1f918d2300 x1853163904599424/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318624.054870:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d2300 x1853163904599424/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318624 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318624.054896:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318624.054924:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318624.054928:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904599296:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767318624.054954:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318624.054960:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318624.054964:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904599424:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767318624.054994:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318624.055001:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767318624.055005:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318624.055012:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3847 00000100:00080000:1.0:1767318624.055023:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318624.055029:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767318624.055060:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0380 x1853163904599552/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318624.055089:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318624.055093:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767318624.055095:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318624.055106:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3847 00000100:00080000:1.0:1767318624.055111:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318624.055114:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767318624.055124:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767318624.055137:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3b80 x1853163904599680/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767318624.055278:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904599552:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767318624.055405:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904599680:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318624.182963:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318624.375182:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318625.209893:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318625.399252:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318626.230937:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318626.422810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318627.258878:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318627.446826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318628.280869:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318628.471711:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318629.176423:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318629.176435:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318629.176438:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318629.176441:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318629.302894:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318629.495164:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318630.328131:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318630.519616:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318631.351410:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318631.544006:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318632.379171:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318632.566902:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318633.399044:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318633.590893:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318634.299180:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318634.299190:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318634.299193:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318634.299196:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318634.423602:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318634.614830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318635.446971:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318635.638933:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318636.470935:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318636.662828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318637.496629:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318637.686893:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318638.518917:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318638.712051:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318639.415165:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318639.415177:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318639.415180:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318639.415183:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318639.543000:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318639.735465:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318640.566898:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318640.758837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318641.592832:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318641.782882:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318642.614906:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318642.806816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318643.640232:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318643.830809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318644.535104:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318644.535114:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318644.535117:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318644.535119:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318644.663177:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318644.856252:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318645.690033:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318645.878831:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318646.712042:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318646.903234:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318647.734908:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318647.926806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318648.759706:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318648.950898:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318649.654896:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318649.654927:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318649.654930:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318649.654934:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767318649.655036:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318624/real 1767318624] req@ffff9d1f918d0380 x1853163904599552/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318649.655076:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0380 x1853163904599552/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318649 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318649.655087:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318624/real 1767318624] req@ffff9d1f918d3b80 x1853163904599680/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318649.655098:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3b80 x1853163904599680/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318649 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318649.655120:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318649.655140:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318649.655144:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904599552:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767318649.655150:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318649.655155:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318649.655158:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904599680:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767318649.655217:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318649.655220:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767318649.655222:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318649.655226:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3871 00000100:00080000:3.0:1767318649.655234:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318649.655240:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767318649.655272:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71180 x1853163904599808/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767318649.655296:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318649.655300:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767318649.655301:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318649.655305:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3871 00000100:00080000:3.0:1767318649.655309:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318649.655313:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767318649.655323:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767318649.655333:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73480 x1853163904599936/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318649.655395:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904599808:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767318649.655504:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904599936:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318649.785324:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318649.974826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318650.806930:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318650.998759:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318651.831372:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318652.022811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318652.854955:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318653.046848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318653.879097:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318654.071157:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318654.774857:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318654.774869:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318654.774873:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318654.774877:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318654.905535:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318655.094810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318655.927173:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318656.119097:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318656.951540:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318657.143179:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318657.978690:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318658.166833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318658.998978:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318659.190828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318659.896697:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318659.896708:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318659.896711:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318659.896714:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318660.022901:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318660.214819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318661.048151:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318661.238893:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318662.071932:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318662.262879:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318663.094993:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318663.286874:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318664.119798:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318664.311580:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318665.015251:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318665.015265:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318665.015268:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318665.015271:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318665.146296:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318665.334866:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318666.166979:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318666.359003:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318667.191130:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318667.383223:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318668.214910:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318668.406832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318669.242715:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318669.430798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318670.137068:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318670.137087:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318670.137091:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318670.137095:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318670.263052:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318670.454920:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318671.287721:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318671.478902:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318672.311069:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318672.502876:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318673.335013:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318673.529215:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318674.230908:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318649/real 1767318649] req@ffff9d1f92f71180 x1853163904599808/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318674.230954:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71180 x1853163904599808/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318674 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318674.230974:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318649/real 1767318649] req@ffff9d1f92f73480 x1853163904599936/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318674.230987:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73480 x1853163904599936/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318674 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318674.231009:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318674.231038:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318674.231070:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904599808:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767318674.231079:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318674.231088:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318674.231093:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904599936:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767318674.231101:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318674.231115:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767318674.231120:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318674.231134:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3897 00000100:00080000:1.0:1767318674.231146:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318674.231154:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767318674.234260:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3b80 x1853163904600064/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318674.234313:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318674.234318:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767318674.234321:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318674.234389:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3897 00000100:00080000:1.0:1767318674.234401:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318674.234407:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767318674.234417:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767318674.234436:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0380 x1853163904600192/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318674.234634:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904600064:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767318674.234883:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904600192:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318674.358884:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318674.550812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318675.385580:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318675.575139:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318676.411236:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318676.599064:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318677.430934:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318677.623227:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318678.456088:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318678.646989:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318679.350851:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318679.350875:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318679.350878:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318679.350882:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318679.480881:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318679.672339:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318680.504056:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318680.695012:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318681.526986:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318681.718786:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318682.551271:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318682.744878:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318683.580159:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318683.767289:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318684.472767:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318684.472779:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318684.472799:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318684.472803:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318684.598922:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318684.790913:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318685.622976:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318685.816810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318686.647533:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318686.838837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318687.673191:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318687.862994:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318688.697466:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318688.886801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318689.593924:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318689.593935:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318689.593939:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318689.593943:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318689.725605:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318689.910834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318690.742931:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318690.934778:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318691.768204:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318691.958830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318692.791164:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318692.982847:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318693.814992:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318694.006820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318694.711730:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318694.711745:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318694.711749:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318694.711752:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318694.838914:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318695.030875:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318695.863916:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318696.054773:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318696.890100:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318697.078979:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318697.917622:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318698.104175:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318698.935243:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318699.128701:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767318699.830847:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318674/real 1767318674] req@ffff9d1f918d3b80 x1853163904600064/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318699 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318699.830906:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3b80 x1853163904600064/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318699 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318699.830917:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318674/real 1767318674] req@ffff9d1f918d0380 x1853163904600192/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318699 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318699.830927:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0380 x1853163904600192/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318699 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767318699.830946:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318699.830977:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318699.830981:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904600064:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767318699.830987:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318699.830993:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318699.830996:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904600192:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767318699.832921:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318699.832932:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767318699.832936:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318699.832945:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3921 00000100:00080000:1.0:1767318699.832956:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318699.832962:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767318699.833012:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d2300 x1853163904600320/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318699.833046:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318699.833050:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767318699.833052:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318699.833056:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3921 00000100:00080000:1.0:1767318699.833059:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318699.833063:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767318699.833082:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767318699.833094:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1500 x1853163904600448/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767318699.833247:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904600320:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767318699.833343:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904600448:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318699.959307:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318700.150825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318700.982927:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318701.174841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318702.006919:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318702.199266:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318703.032358:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318703.222824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318704.055139:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318704.248734:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318704.954754:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318704.954767:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318704.954770:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318704.954773:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318705.078974:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318705.270841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318706.103411:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318706.294890:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318707.132086:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318707.319002:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318708.153160:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318708.342970:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318709.174955:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318709.370182:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318710.070863:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318710.070873:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318710.070875:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318710.070879:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318710.199527:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318710.390866:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318711.223296:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318711.414960:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318712.246897:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318712.439673:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318713.270880:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318713.462806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318714.294916:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318714.486805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318715.195415:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318715.195424:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318715.195444:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318715.195448:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318715.318896:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318715.510762:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318716.342908:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318716.535754:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318717.366993:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318717.558772:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318718.391555:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318718.587101:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318719.419081:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318719.606780:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318720.314805:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318720.314816:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318720.314819:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318720.314822:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318720.439546:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318720.630801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318721.464239:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318721.654835:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318722.490028:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318722.679457:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318723.510905:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318723.705007:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767318724.471697:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318699/real 1767318699] req@ffff9d1f918d2300 x1853163904600320/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318724.471734:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d2300 x1853163904600320/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318724 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318724.471747:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318699/real 1767318699] req@ffff9d1f918d1500 x1853163904600448/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318724.471759:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1500 x1853163904600448/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318724 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767318724.471778:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318724.471787:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318724.471808:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904600320:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767318724.471815:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318724.471821:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318724.471824:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904600448:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767318724.476152:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318724.476162:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767318724.476166:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767318724.476175:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3947 00000100:00080000:2.0:1767318724.476186:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767318724.476193:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767318724.476231:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cce00 x1853163904600576/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767318724.476249:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318724.476253:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767318724.476255:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767318724.476258:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3947 00000100:00080000:2.0:1767318724.476272:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767318724.476276:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767318724.476285:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767318724.476296:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cc700 x1853163904600704/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767318724.476540:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904600576:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767318724.476645:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904600704:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318724.539235:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318724.729618:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318725.560244:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318725.750848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318726.582960:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318726.775127:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318727.608328:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318727.798909:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318728.631489:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318728.828243:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318729.526846:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318729.526863:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318729.526867:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318729.526871:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318729.655069:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318729.846870:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318730.679057:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318730.870978:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318731.703739:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318731.897574:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318732.727039:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318732.918939:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318733.755895:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318733.942815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318734.647280:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318734.647291:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318734.647294:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318734.647297:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318734.775517:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318734.966844:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318735.800153:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318735.990803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318736.822972:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318737.016243:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318737.847096:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318738.040476:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318738.871309:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318739.062826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318739.766772:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318739.766783:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318739.766786:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318739.766789:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318739.895039:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318740.089255:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318740.918923:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318741.110730:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318741.942993:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318742.134855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318742.971084:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318743.164754:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318743.991521:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318744.183094:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318744.886880:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318744.886891:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318744.886894:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318744.886898:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318745.015160:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318745.206972:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318746.038992:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318746.230833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318747.064900:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318747.254858:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318748.087868:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318748.279762:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318749.110962:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318749.302817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767318750.006829:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318724/real 1767318724] req@ffff9d1f967cce00 x1853163904600576/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318750.006865:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cce00 x1853163904600576/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318749 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318750.006899:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318724/real 1767318724] req@ffff9d1f967cc700 x1853163904600704/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318750.006912:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cc700 x1853163904600704/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318749 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767318750.006930:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767318750.007008:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767318750.007012:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904600576:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767318750.007040:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318750.015805:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318750.015816:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767318750.015820:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318750.015831:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3972 00000100:00080000:0.0:1767318750.015842:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318750.015849:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767318750.015889:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031880 x1853163904600832/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767318750.016001:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318750.016005:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767318750.016253:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318750.016267:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3972 00000100:00080000:0.0:1767318750.016271:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318750.016277:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767318750.016292:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767318750.016306:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032d80 x1853163904600960/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318750.016337:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767318750.016350:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904600704:192.168.202.140@tcp:38:kworker.0 00000100:00100000:1.0:1767318750.016389:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904600832:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767318750.016477:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904600960:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318750.134997:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318750.328188:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318751.159052:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318751.350774:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318752.183886:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318752.374801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318753.209958:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318753.399876:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318754.234155:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318754.422814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318755.128407:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318755.128418:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318755.128421:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318755.128424:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318755.256244:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318755.446806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318756.279052:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318756.470809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318757.302838:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318757.494805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318758.327010:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318758.518808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318759.352593:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318759.542847:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318760.253971:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318760.253982:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318760.253985:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318760.253988:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318760.375432:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318760.567212:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318761.398943:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318761.591057:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318762.423832:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318762.615053:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318763.447251:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318763.638804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318764.472636:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318764.662794:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318765.367810:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318765.367820:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318765.367822:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318765.367825:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318765.496701:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318765.688802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318766.518921:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318766.710803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318767.543451:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318767.735018:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318768.566926:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318768.758954:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318769.591032:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318769.784644:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318770.487195:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318770.487206:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318770.487209:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318770.487211:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318770.615205:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318770.806876:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318771.638933:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318771.830777:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318772.662939:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318772.854768:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318773.687301:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318773.880362:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318774.712306:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318774.902843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767318775.606855:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318750/real 1767318750] req@ffff9d1f91031880 x1853163904600832/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318775.606895:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031880 x1853163904600832/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318775 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318775.606905:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318750/real 1767318750] req@ffff9d1f91032d80 x1853163904600960/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318775.606916:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032d80 x1853163904600960/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318775 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767318775.606966:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767318775.606996:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767318775.607001:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904600832:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767318775.607008:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767318775.607013:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767318775.607016:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904600960:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767318775.607188:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318775.607194:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767318775.607198:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767318775.607204:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 3997 00000100:00080000:2.0:1767318775.607216:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767318775.607223:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767318775.607261:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cea00 x1853163904601088/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767318775.607290:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318775.607293:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767318775.607295:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767318775.607298:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 3997 00000100:00080000:2.0:1767318775.607302:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767318775.607305:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767318775.607317:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767318775.607328:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cd500 x1853163904601216/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767318775.607618:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904601088:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767318775.607764:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904601216:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318775.735278:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318775.926851:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318776.758996:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318776.954784:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318777.783180:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318777.974965:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318778.808326:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318779.001711:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318779.831554:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318780.024743:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318780.726843:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318780.726854:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318780.726875:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318780.726878:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318780.857062:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318781.047207:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318781.879082:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318782.070809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318782.903009:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318783.095050:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318783.927180:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318784.119413:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318784.951170:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318785.143039:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318785.851370:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318785.851383:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318785.851385:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318785.851389:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318785.979322:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318786.166919:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318786.998984:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318787.190948:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318788.023528:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318788.215185:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318789.046910:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318789.241395:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318790.071056:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318790.267810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318790.966824:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318790.966836:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318790.966839:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318790.966842:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318791.094983:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318791.287383:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318792.119364:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318792.310843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318793.149321:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318793.338028:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318794.167010:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318794.358808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318795.190936:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318795.382806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318796.087208:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318796.087219:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318796.087222:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318796.087225:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318796.216787:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318796.406820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318797.238993:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318797.430965:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318798.265240:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318798.454815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318799.287226:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318799.478998:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767318800.182813:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318775/real 1767318775] req@ffff9d1f967cea00 x1853163904601088/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318800 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318800.182849:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cea00 x1853163904601088/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318800 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318800.182858:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318775/real 1767318775] req@ffff9d1f967cd500 x1853163904601216/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318800 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318800.182890:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cd500 x1853163904601216/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318800 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767318800.182906:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318800.182937:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318800.182941:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904601088:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767318800.182947:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318800.182952:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318800.182955:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904601216:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767318800.183116:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318800.183122:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767318800.183125:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318800.183130:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4023 00000100:00080000:0.0:1767318800.183138:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318800.183156:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767318800.183188:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030380 x1853163904601344/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767318800.183226:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318800.183230:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767318800.183232:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318800.183236:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4023 00000100:00080000:0.0:1767318800.183240:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318800.183245:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767318800.183255:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767318800.183267:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033100 x1853163904601472/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767318800.183696:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904601344:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767318800.183781:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904601472:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767318800.311663:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318800.502888:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318801.336662:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318801.526795:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318802.359323:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318802.550807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318803.383138:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318803.574821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318804.407793:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318804.598814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318805.302867:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318805.302878:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318805.302881:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318805.302884:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318805.432238:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318805.624186:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318806.456322:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318806.646809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318807.481597:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318807.672033:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318808.503788:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318808.694817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318809.527882:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318809.718823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318810.422818:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318810.422829:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318810.422832:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318810.422837:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318810.552719:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318810.742815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318811.575265:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318811.766831:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318812.601335:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318812.790821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318813.622995:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318813.814871:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318814.646898:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318814.838843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318815.544251:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318815.544263:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318815.544267:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318815.544270:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318815.671741:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318815.862798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318816.697705:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318816.886802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318817.718898:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318817.915144:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318818.743151:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318818.935107:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318819.774105:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318819.964351:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318820.662819:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318820.662829:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318820.662832:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318820.662836:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318820.794709:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318820.982812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318821.814961:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318822.012744:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318822.841103:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318823.031045:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318823.867159:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318824.054947:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318824.888921:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318825.078844:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767318825.782834:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318800/real 1767318800] req@ffff9d1f91030380 x1853163904601344/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318825 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318825.782870:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030380 x1853163904601344/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318825 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318825.782901:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318800/real 1767318800] req@ffff9d1f91033100 x1853163904601472/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318825 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318825.782915:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033100 x1853163904601472/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318825 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767318825.782934:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767318825.782963:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767318825.782969:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904601344:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767318825.782975:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767318825.782981:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767318825.782985:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904601472:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767318825.783164:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318825.783170:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767318825.783173:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318825.783179:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4047 00000100:00080000:0.0:1767318825.783190:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318825.783204:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767318825.783248:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033480 x1853163904601600/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767318825.783270:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318825.783273:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767318825.783275:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318825.783279:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4047 00000100:00080000:0.0:1767318825.783283:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318825.783286:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767318825.783295:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767318825.783305:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030700 x1853163904601728/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767318825.783465:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904601600:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767318825.783576:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904601728:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318825.911456:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318826.103283:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318826.934865:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318827.127731:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318827.961385:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318828.150798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318828.982922:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318829.174823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318830.007941:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318830.198850:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318830.903325:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318830.903340:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318830.903343:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318830.903347:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318831.030948:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318831.223791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318832.055550:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318832.249439:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318833.082962:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318833.274479:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318834.103004:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318834.297769:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318835.127317:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318835.318859:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318836.022902:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318836.022915:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318836.022918:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318836.022921:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318836.150929:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318836.343169:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318837.174899:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318837.366799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318838.203432:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318838.390799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318839.223645:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318839.415328:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318840.246982:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318840.438811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318841.142849:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318841.142863:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318841.142866:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318841.142895:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318841.270902:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318841.463137:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318842.298478:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318842.487378:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318843.319889:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318843.511398:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318844.342940:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318844.534930:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318845.368468:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318845.562738:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318846.263106:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318846.263116:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318846.263120:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318846.263123:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318846.390939:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318846.582812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318847.414941:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318847.606881:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318848.439255:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318848.630873:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318849.462892:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318849.654888:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767318850.358929:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318825/real 1767318825] req@ffff9d1f91033480 x1853163904601600/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318850 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318850.358965:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033480 x1853163904601600/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318850 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318850.358976:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318825/real 1767318825] req@ffff9d1f91030700 x1853163904601728/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318850 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318850.358988:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030700 x1853163904601728/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318850 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767318850.359007:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767318850.359086:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767318850.359092:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904601600:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767318850.359099:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318850.359149:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318850.359155:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767318850.359175:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318850.359180:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4073 00000100:00080000:3.0:1767318850.359190:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318850.359196:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767318850.359240:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71f80 x1853163904601856/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767318850.359255:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318850.359259:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767318850.359260:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318850.359263:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4073 00000100:00080000:3.0:1767318850.359268:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318850.359272:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767318850.359281:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767318850.359290:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72680 x1853163904601984/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318850.361424:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767318850.361436:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904601728:192.168.202.140@tcp:38:kworker.0 00000100:00100000:1.0:1767318850.361492:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904601856:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767318850.361603:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904601984:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318850.487242:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318850.678827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318851.516475:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318851.702843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318852.539386:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318852.727241:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318853.559139:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318853.750819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318854.583029:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318854.774908:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318855.480183:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318855.480195:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318855.480198:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318855.480201:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318855.607181:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318855.798896:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318856.631333:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318856.824858:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318857.654905:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318857.847650:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318858.679079:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318858.870849:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318859.703049:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318859.894806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318860.599063:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318860.599091:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318860.599093:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318860.599096:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318860.726894:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318860.918798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318861.751095:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318861.942774:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318862.775502:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318862.968149:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318863.799437:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318863.990868:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318864.823006:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318865.014816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318865.719713:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318865.719724:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318865.719727:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318865.719730:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318865.846908:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318866.038854:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318866.870924:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318867.062798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318867.894918:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318868.086814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318868.918971:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318869.110792:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318869.944047:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318870.134829:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318870.839014:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318870.839033:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318870.839035:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318870.839039:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318870.966904:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318871.161174:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318871.991117:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318872.182802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318873.015168:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318873.206874:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318874.039658:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318874.230951:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318875.062939:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318875.255283:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318875.958815:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318875.958827:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318875.958830:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318875.958834:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1767318875.960237:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318850/real 1767318850] req@ffff9d1f92f71f80 x1853163904601856/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318875.960279:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71f80 x1853163904601856/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318875 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318875.960290:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318850/real 1767318850] req@ffff9d1f92f72680 x1853163904601984/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767318875.960304:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72680 x1853163904601984/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318875 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767318875.960327:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767318875.960358:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767318875.960363:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904601856:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767318875.960371:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767318875.960377:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767318875.960381:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904601984:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767318875.961608:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318875.961615:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767318875.961619:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318875.961627:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4098 00000100:00080000:3.0:1767318875.961636:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318875.961661:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767318875.961720:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72d80 x1853163904602112/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767318875.961748:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318875.961752:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767318875.961753:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767318875.961757:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4098 00000100:00080000:3.0:1767318875.961760:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767318875.961764:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767318875.961786:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767318875.961799:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72a00 x1853163904602240/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767318875.961938:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904602112:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767318875.962053:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904602240:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767318876.087577:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318876.278899:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318877.110836:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318877.303292:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318878.136605:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767318878.326804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318879.159163:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318879.353373:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318880.184905:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318880.374800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318881.078792:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318881.078804:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318881.078807:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318881.078810:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318881.208794:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318881.398858:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318882.231514:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318882.423265:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318883.258879:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318883.446790:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318884.279013:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318884.471291:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318885.305610:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318885.494867:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318886.199008:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318886.199021:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318886.199024:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318886.199028:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318886.326908:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318886.518818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318887.350903:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318887.542825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318888.375234:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318888.566817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318889.401170:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318889.590845:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318890.425217:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318890.614928:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318891.322324:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318891.322336:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318891.322339:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318891.322342:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318891.452842:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318891.638817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318892.471055:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318892.662988:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318893.494958:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318893.686856:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318894.518932:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318894.710836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318895.543648:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318895.734943:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318896.439035:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318896.439047:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318896.439050:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318896.439052:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318896.566957:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318896.758984:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318897.590971:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318897.782867:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318898.615042:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318898.807044:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318899.639150:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318899.831864:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767318900.535108:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318875/real 1767318875] req@ffff9d1f92f72d80 x1853163904602112/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318900 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318900.535165:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72d80 x1853163904602112/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318900 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318900.535189:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318875/real 1767318875] req@ffff9d1f92f72a00 x1853163904602240/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318900 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767318900.535211:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72a00 x1853163904602240/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318900 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767318900.535231:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767318900.535326:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318900.535331:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904602112:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767318900.535347:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318900.535988:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318900.535995:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767318900.535999:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318900.536003:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4123 00000100:00080000:0.0:1767318900.536013:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318900.536019:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767318900.536051:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033b80 x1853163904602368/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767318900.536066:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318900.536069:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767318900.536071:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318900.536074:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4123 00000100:00080000:0.0:1767318900.536077:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318900.536080:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767318900.536089:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767318900.536098:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033800 x1853163904602496/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767318900.536493:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767318900.536499:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904602240:192.168.202.140@tcp:38:kworker.0 00000100:00100000:2.0:1767318900.536531:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904602368:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767318900.536642:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904602496:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767318900.665174:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318900.856538:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318901.696595:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318901.878829:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318902.714288:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318902.902791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318903.735011:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318903.926949:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318904.758955:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318904.950836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318905.654958:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318905.654969:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318905.654972:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318905.654991:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318905.783104:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318905.974793:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767318906.807470:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318906.999873:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318907.831059:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318908.023558:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318908.863502:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318909.046819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318909.879170:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318910.070848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318910.779027:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318910.779039:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318910.779043:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318910.779047:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767318910.903079:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318911.094805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318911.928474:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318912.119064:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318912.950938:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318913.142833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318913.975056:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318914.169138:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318914.998949:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318915.191244:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318915.895695:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318915.895706:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318915.895709:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318915.895712:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318916.022924:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318916.218751:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318917.047086:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318917.238816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318918.070909:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318918.262860:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318919.096120:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318919.287276:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318920.119133:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318920.311848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318921.014972:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318921.014982:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318921.014985:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318921.014988:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318921.142932:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318921.334877:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318922.169761:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318922.358813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318923.191646:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318923.382838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318924.215285:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318924.407114:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767318925.110832:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318900/real 1767318900] req@ffff9d1f91033b80 x1853163904602368/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318925.110871:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033b80 x1853163904602368/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318925 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318925.110883:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318900/real 1767318900] req@ffff9d1f91033800 x1853163904602496/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318925.110895:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033800 x1853163904602496/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318925 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767318925.110923:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318925.110944:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318925.110948:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904602368:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767318925.110953:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318925.110958:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318925.110961:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904602496:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767318925.111089:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318925.111095:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767318925.111099:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767318925.111105:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4148 00000100:00080000:2.0:1767318925.111118:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767318925.111125:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767318925.111160:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967ce300 x1853163904602624/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318925.111252:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904602624:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767318925.111576:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318925.111580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767318925.111583:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767318925.111587:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4148 00000100:00080000:2.0:1767318925.111591:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767318925.111595:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767318925.111604:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767318925.111618:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cdf80 x1853163904602752/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318925.113258:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904602752:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767318925.239006:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318925.430804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318926.262945:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318926.454818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318927.287228:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318927.478892:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318928.313467:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318928.502815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318929.339511:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318929.526791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318930.230825:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318930.230836:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318930.230839:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318930.230843:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318930.360852:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318930.550805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318931.384838:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318931.574866:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318932.406930:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318932.598814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318933.430950:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318933.622812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318934.455890:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318934.646954:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318935.351601:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318935.351611:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318935.351614:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318935.351616:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318935.478993:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318935.670814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318936.503065:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318936.694913:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318937.530056:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318937.718804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318938.551491:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318938.743479:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318939.580414:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318939.767750:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318940.475722:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318940.475733:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318940.475737:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318940.475740:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318940.598895:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318940.793382:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318941.622951:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318941.814884:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318942.650587:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318942.838843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318943.671120:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318943.862967:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318944.695213:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318944.886833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318945.594633:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318945.594643:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318945.594647:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318945.594669:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318945.718886:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318945.911561:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318946.742995:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318946.935108:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318947.770587:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318947.961417:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318948.790918:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318948.982827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318949.824659:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318950.006818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767318950.710909:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318925/real 1767318925] req@ffff9d1f967cdf80 x1853163904602752/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318950 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318950.710945:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cdf80 x1853163904602752/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318950 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318950.710954:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318925/real 1767318925] req@ffff9d1f967ce300 x1853163904602624/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318950 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767318950.710966:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967ce300 x1853163904602624/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318950 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767318950.710986:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318950.710997:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318950.711000:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904602752:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767318950.711005:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767318950.711021:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767318950.711023:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904602624:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767318950.711148:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318950.711154:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767318950.711157:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318950.711162:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4172 00000100:00080000:1.0:1767318950.711171:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318950.711177:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767318950.711225:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1180 x1853163904602880/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767318950.711269:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318950.711273:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767318950.711275:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767318950.711279:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4172 00000100:00080000:1.0:1767318950.711283:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767318950.711287:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767318950.711296:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767318950.711309:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3480 x1853163904603008/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767318950.711555:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904602880:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767318950.711860:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904603008:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318950.839424:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318951.030795:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318951.864961:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318952.054827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318952.889328:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318953.081309:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318953.911270:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318954.102822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318954.935482:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318955.126885:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767318955.834313:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318955.834323:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767318955.834326:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767318955.834330:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318955.959260:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318956.153220:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318956.982905:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318957.174824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318958.007728:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318958.198972:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318959.031849:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318959.222787:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318960.054960:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318960.247738:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767318960.950869:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318960.950879:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767318960.950881:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767318960.950885:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318961.079201:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318961.271222:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318962.103596:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318962.294943:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318963.127067:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318963.318829:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318964.150966:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318964.342798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318965.175637:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318965.371198:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318966.071172:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318966.071182:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318966.071185:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318966.071189:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318966.200747:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318966.390890:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318967.223443:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318967.416294:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318968.249400:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318968.438852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318969.270939:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318969.462837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318970.295091:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318970.487169:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318971.191458:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318971.191469:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318971.191472:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318971.191475:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318971.318966:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318971.510794:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318972.344964:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318972.535724:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318973.366923:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318973.558798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318974.392315:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318974.582812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767318975.286819:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318950/real 1767318950] req@ffff9d1f918d1180 x1853163904602880/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318975 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318975.286854:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1180 x1853163904602880/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767318975 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318975.286864:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318950/real 1767318950] req@ffff9d1f918d3480 x1853163904603008/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318975 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767318975.286874:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3480 x1853163904603008/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767318975 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767318975.286893:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318975.286922:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318975.286927:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904602880:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767318975.286933:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767318975.286939:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767318975.286942:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904603008:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767318975.287008:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318975.287013:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767318975.287017:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318975.287022:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4198 00000100:00080000:0.0:1767318975.287031:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318975.287037:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767318975.287070:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031c00 x1853163904603136/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767318975.287096:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318975.287113:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767318975.287115:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767318975.287119:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4198 00000100:00080000:0.0:1767318975.287124:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767318975.287128:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767318975.287138:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767318975.287151:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031f80 x1853163904603264/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767318975.287156:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904603136:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767318975.287267:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904603264:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767318975.415210:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318975.606902:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318976.439625:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318976.631572:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318977.464158:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318977.654805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318978.486990:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318978.678818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318979.511205:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318979.706327:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318980.407289:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318980.407299:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318980.407302:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318980.407305:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318980.534904:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318980.726871:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318981.558939:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318981.751106:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318982.585255:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318982.775146:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318983.606882:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318983.798812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318984.633099:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318984.824723:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318985.527639:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318985.527649:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318985.527652:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318985.527656:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318985.655061:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318985.846920:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318986.680352:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318986.870816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318987.704497:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318987.894814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318988.727414:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318988.918887:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318989.751069:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318989.945292:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767318990.647152:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318990.647165:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767318990.647169:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767318990.647172:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767318990.777071:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318990.967505:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318991.798923:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318991.992223:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318992.822891:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318993.015128:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318993.855946:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318994.038934:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318994.871230:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318995.062928:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767318995.766881:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318995.766899:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767318995.766903:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767318995.766907:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767318995.894920:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318996.088493:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318996.920410:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318997.111119:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318997.943633:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318998.135427:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767318998.966975:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767318999.160214:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767318999.991979:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319000.187181:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319000.886911:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319000.886925:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319000.886960:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319000.886964:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767319000.888526:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318975/real 1767318975] req@ffff9d1f91031f80 x1853163904603264/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319000 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319000.888571:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031f80 x1853163904603264/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319000 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319000.888584:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318975/real 1767318975] req@ffff9d1f91031c00 x1853163904603136/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319000 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319000.888596:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031c00 x1853163904603136/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319000 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767319000.888620:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319000.888642:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319000.888648:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904603264:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767319000.888654:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319000.888660:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319000.888663:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904603136:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319000.888846:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319000.888854:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319000.888858:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319000.888866:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4222 00000100:00080000:0.0:1767319000.888880:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319000.888891:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767319000.888945:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030e00 x1853163904603392/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319000.889013:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904603392:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319000.889172:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319000.889176:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319000.889179:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319000.889184:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4222 00000100:00080000:0.0:1767319000.889188:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319000.889193:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767319000.889204:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319000.889237:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032a00 x1853163904603520/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319000.894446:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904603520:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767319001.015060:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319001.206826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319002.038965:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319002.230854:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319003.063873:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319003.255414:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319004.087735:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319004.281559:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319005.113014:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319005.302895:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319006.011042:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319006.011058:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319006.011062:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319006.011065:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319006.136563:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319006.329545:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319007.158943:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319007.351610:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319008.184859:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319008.374845:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319009.207303:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319009.398806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319010.230996:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319010.423180:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319011.126797:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319011.126809:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319011.126812:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319011.126815:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319011.255292:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319011.446892:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319012.280231:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319012.470941:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319013.302951:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319013.494851:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319014.327320:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319014.518836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319015.351335:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319015.542814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319016.246826:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319016.246838:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319016.246841:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319016.246845:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319016.375795:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319016.566808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319017.401266:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319017.591938:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319018.423501:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319018.614857:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319019.447294:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319019.638912:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319020.471064:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319020.663743:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319021.366842:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319021.366854:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319021.366857:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319021.366860:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319021.495103:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319021.686794:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319022.519109:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319022.710806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319023.543831:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319023.734798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319024.566973:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319024.758787:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767319025.462829:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319000/real 1767319000] req@ffff9d1f91032a00 x1853163904603520/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319025 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319025.462874:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032a00 x1853163904603520/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319025 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319025.462886:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319000/real 1767319000] req@ffff9d1f91030e00 x1853163904603392/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319025 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319025.462898:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030e00 x1853163904603392/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319025 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767319025.462919:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767319025.463014:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767319025.463019:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904603520:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767319025.463025:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319025.463053:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319025.463058:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319025.463061:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319025.463066:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4248 00000100:00080000:0.0:1767319025.463075:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319025.463081:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767319025.463116:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031c00 x1853163904603648/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319025.463164:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319025.463167:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319025.463168:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319025.463172:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4248 00000100:00080000:0.0:1767319025.463176:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319025.463180:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767319025.463190:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319025.463201:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031f80 x1853163904603776/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319025.463975:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767319025.463982:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904603392:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767319025.464017:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904603648:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767319025.464191:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904603776:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767319025.590870:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319025.782809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319026.615285:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319026.807329:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319027.639217:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319027.830815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319028.662940:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319028.856715:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319029.686943:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319029.883624:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319030.588034:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319030.588045:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319030.588048:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319030.588069:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319030.711284:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319030.902812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319031.737290:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319031.926813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319032.760028:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319032.950878:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319033.782949:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319033.975031:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319034.807079:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319034.999147:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319035.702949:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319035.702960:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319035.702964:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319035.702967:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319035.831065:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319036.022830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319036.855208:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319037.046993:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319037.879346:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319038.070928:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319038.904182:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319039.095733:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319039.927118:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319040.118830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319040.823452:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319040.823462:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319040.823465:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319040.823468:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319040.955787:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319041.142807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319041.974972:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319042.166870:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319042.998937:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319043.191064:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319044.023003:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319044.215375:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319045.048864:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319045.239120:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319045.942861:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319045.942875:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319045.942878:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319045.942908:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319046.078607:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319046.262861:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319047.099236:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319047.290221:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319048.120948:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319048.311397:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319049.145840:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319049.335223:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767319050.039253:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319025/real 1767319025] req@ffff9d1f91031c00 x1853163904603648/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319050.039306:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031c00 x1853163904603648/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319050 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319050.039319:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319025/real 1767319025] req@ffff9d1f91031f80 x1853163904603776/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319050.039332:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031f80 x1853163904603776/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319050 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767319050.039356:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319050.039383:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319050.039387:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904603648:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767319050.039394:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319050.039400:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319050.039403:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904603776:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767319050.039473:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319050.039480:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319050.039484:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319050.039489:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4273 00000100:00080000:0.0:1767319050.039500:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319050.039508:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767319050.039560:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033800 x1853163904603904/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319050.039583:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319050.039586:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319050.039588:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319050.039592:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4273 00000100:00080000:0.0:1767319050.039596:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319050.039600:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767319050.039611:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319050.039622:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033b80 x1853163904604032/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319050.039737:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904603904:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767319050.039862:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904604032:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767319050.167034:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319050.359028:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319051.192255:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319051.383143:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319052.215064:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319052.408898:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319053.239130:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319053.430841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319054.263807:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319054.454860:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319055.160807:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319055.160822:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319055.160826:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319055.160830:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319055.287436:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319055.478814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319056.312365:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319056.502856:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319057.335132:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319057.527104:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319058.362720:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319058.551490:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319059.383064:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319059.575335:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319060.279332:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319060.279347:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319060.279351:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319060.279355:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319060.407978:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319060.599713:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319061.432025:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319061.622849:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319062.455541:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319062.646992:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319063.479737:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319063.672662:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319064.506026:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319064.695521:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319065.399832:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319065.399873:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319065.399877:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319065.399881:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319065.528073:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319065.718864:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319066.554770:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319066.743820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319067.578491:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319067.767703:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319068.598930:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319068.791410:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319069.623114:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319069.815643:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319070.519321:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319070.519333:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319070.519337:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319070.519340:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319070.649566:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319070.840351:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319071.671176:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319071.862799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319072.696969:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319072.886900:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319073.718955:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319073.910812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319074.752840:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319074.934802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767319075.638809:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319050/real 1767319050] req@ffff9d1f91033800 x1853163904603904/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319075 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319075.638844:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033800 x1853163904603904/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319075 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319075.638854:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319050/real 1767319050] req@ffff9d1f91033b80 x1853163904604032/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319075 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319075.638865:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033b80 x1853163904604032/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319075 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767319075.638883:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319075.638906:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319075.638910:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904603904:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319075.638916:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319075.638921:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319075.638924:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904604032:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767319075.639060:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319075.639101:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767319075.639104:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319075.639109:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4297 00000100:00080000:2.0:1767319075.639120:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319075.639126:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767319075.639156:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cf100 x1853163904604160/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767319075.639181:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319075.639199:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767319075.639201:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319075.639204:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4297 00000100:00080000:2.0:1767319075.639209:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319075.639212:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767319075.639222:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767319075.639248:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cdc00 x1853163904604288/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767319075.639307:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904604160:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767319075.639412:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904604288:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767319075.773435:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319075.960649:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319076.794462:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319076.982855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319077.815094:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319078.007440:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319078.838911:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319079.030910:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319079.863674:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319080.054868:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319080.758958:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319080.758970:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319080.758973:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319080.758976:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319080.887139:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319081.078861:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319081.912426:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319082.102871:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319082.935037:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319083.126828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319083.959057:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319084.150852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319084.982996:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319085.174867:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319085.878956:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319085.878968:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319085.878979:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319085.878982:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319086.008184:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319086.201302:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319087.031832:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319087.222819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319088.055224:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319088.246810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319089.078981:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319089.270851:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319090.102912:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319090.295540:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319090.999051:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319090.999061:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319090.999064:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319090.999068:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319091.128447:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319091.318838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319092.151496:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319092.342779:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319093.175467:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319093.368146:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319094.199249:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319094.392182:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319095.222933:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319095.417205:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319096.121847:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319096.121858:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319096.121861:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319096.121864:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319096.246924:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319096.438999:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319097.272225:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319097.462804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319098.295019:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319098.486797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319099.318903:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319099.510827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767319100.214788:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319075/real 1767319075] req@ffff9d1f967cf100 x1853163904604160/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319100.214853:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cf100 x1853163904604160/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319100 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319100.214865:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319075/real 1767319075] req@ffff9d1f967cdc00 x1853163904604288/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319100.214874:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cdc00 x1853163904604288/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319100 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767319100.214889:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319100.214904:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319100.214909:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904604160:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767319100.214915:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319100.214921:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319100.214924:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904604288:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767319100.214929:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319100.214934:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319100.214937:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319100.214942:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4323 00000100:00080000:0.0:1767319100.214952:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319100.214958:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767319100.214989:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033b80 x1853163904604416/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319100.215070:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319100.215074:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319100.215076:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319100.215080:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4323 00000100:00080000:0.0:1767319100.215084:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319100.215088:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767319100.215097:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319100.215109:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033800 x1853163904604544/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319100.215116:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904604416:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767319100.215273:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904604544:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767319100.345571:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319100.534799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319101.367413:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319101.558791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319102.390921:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319102.582809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319103.415124:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319103.607157:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319104.441497:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319104.630824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319105.335378:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319105.335388:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319105.335391:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319105.335393:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319105.463049:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319105.656999:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319106.487448:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319106.678751:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319107.511348:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319107.702861:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319108.534982:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319108.727625:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319109.560598:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319109.751875:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319110.456921:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319110.456931:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319110.456935:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319110.456938:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319110.582944:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319110.776725:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319111.608183:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319111.800086:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319112.630969:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319112.822822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319113.656398:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319113.846861:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319114.678937:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319114.872964:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319115.577902:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319115.577914:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319115.577917:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319115.577921:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319115.703238:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319115.894809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319116.727018:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319116.918797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319117.750927:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319117.943229:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319118.775463:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319118.966883:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319119.798926:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319119.991322:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319120.695362:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319120.695372:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319120.695375:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319120.695379:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319120.822972:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319121.014893:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319121.851034:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319122.039839:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319122.870933:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319123.062837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319123.894878:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319124.086872:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319124.918990:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319125.110792:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767319125.814787:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319100/real 1767319100] req@ffff9d1f91033800 x1853163904604544/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319125.814825:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033800 x1853163904604544/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319125 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319125.814837:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319100/real 1767319100] req@ffff9d1f91033b80 x1853163904604416/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319125.814876:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033b80 x1853163904604416/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319125 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767319125.814895:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319125.814927:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319125.814931:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904604544:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767319125.814937:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319125.814943:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319125.814946:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904604416:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319125.815222:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319125.815227:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319125.815231:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319125.815237:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4347 00000100:00080000:0.0:1767319125.815246:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319125.815252:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767319125.815284:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030700 x1853163904604672/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319125.815308:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319125.815317:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319125.815318:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319125.815322:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4347 00000100:00080000:0.0:1767319125.815325:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319125.815328:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767319125.815336:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319125.815347:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033480 x1853163904604800/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319125.815485:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904604672:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767319125.815582:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904604800:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767319125.944217:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319126.134797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319126.966959:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319127.159304:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319127.991209:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319128.182846:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319129.016760:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319129.206780:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319130.039216:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319130.231260:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319130.935134:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319130.935145:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319130.935147:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319130.935151:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319131.062957:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319131.254860:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319132.087014:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319132.279406:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319133.111108:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319133.306582:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319134.134980:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319134.326808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319135.159003:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319135.350761:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319136.055020:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319136.055030:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319136.055033:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319136.055054:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319136.183189:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319136.375069:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319137.207625:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319137.399099:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319138.231227:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319138.423025:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319139.255374:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319139.447565:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319140.278849:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319140.471148:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319141.174981:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319141.174990:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319141.174992:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319141.174995:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319141.302960:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319141.494792:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319142.326986:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319142.518800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319143.357232:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319143.542796:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319144.375092:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319144.571893:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319145.403565:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319145.592752:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319146.294852:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319146.294864:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319146.294866:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319146.294869:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319146.425723:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319146.614857:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319147.447474:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319147.638779:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319148.471238:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319148.662804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319149.495081:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319149.689739:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767319150.393768:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319125/real 1767319125] req@ffff9d1f91030700 x1853163904604672/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319150 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319150.393799:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030700 x1853163904604672/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319150 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319150.393806:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319125/real 1767319125] req@ffff9d1f91033480 x1853163904604800/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319150 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319150.393814:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033480 x1853163904604800/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319150 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767319150.393827:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319150.393842:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319150.393862:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904604672:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319150.393866:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319150.393870:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319150.393872:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904604800:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767319150.398062:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319150.398094:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767319150.398099:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319150.398111:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4373 00000100:00080000:2.0:1767319150.398122:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319150.398130:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767319150.398180:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cdc00 x1853163904604928/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767319150.398211:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319150.398215:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767319150.398217:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319150.398221:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4373 00000100:00080000:2.0:1767319150.398225:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319150.398230:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767319150.398240:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319150.398243:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904604928:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767319150.398253:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cf100 x1853163904605056/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767319150.398359:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904605056:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767319150.519103:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319150.710836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319151.550013:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319151.734844:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319152.567111:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319152.758791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319153.595579:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319153.782798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319154.616851:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319154.806800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319155.510938:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319155.510948:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319155.510951:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319155.510954:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319155.639830:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319155.830822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319156.662887:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319156.854930:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319157.691615:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319157.878758:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319158.715226:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319158.902808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319159.735007:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319159.926823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319160.633066:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319160.633076:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319160.633079:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319160.633083:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319160.758971:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319160.950904:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319161.783155:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319161.975973:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319162.807049:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319162.998861:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319163.831570:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319164.023072:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319164.855336:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319165.046928:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319165.750913:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319165.750924:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319165.750926:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319165.750946:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319165.878892:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319166.070856:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319166.903496:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319167.095403:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319167.927334:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319168.119507:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319168.950909:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319169.142849:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319169.975119:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319170.166754:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319170.870906:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319170.870917:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319170.870919:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319170.870923:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319170.998959:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319171.190815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319172.024817:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319172.214900:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319173.050347:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319173.238811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319174.070998:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319174.262807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319175.094911:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319175.286906:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767319175.991008:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319150/real 1767319150] req@ffff9d1f967cf100 x1853163904605056/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319175 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319175.991060:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cf100 x1853163904605056/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319175 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319175.991070:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319150/real 1767319150] req@ffff9d1f967cdc00 x1853163904604928/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319175 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319175.991083:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cdc00 x1853163904604928/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319175 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767319175.991102:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319175.991129:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319175.991134:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904605056:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767319175.991168:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319175.991175:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319175.991179:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904604928:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767319175.991267:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319175.991273:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767319175.991277:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767319175.991282:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4398 00000100:00080000:1.0:1767319175.991292:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767319175.991298:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767319175.991329:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d0700 x1853163904605184/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767319175.991371:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319175.991374:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767319175.991376:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767319175.991380:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4398 00000100:00080000:1.0:1767319175.991384:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767319175.991388:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767319175.991398:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767319175.991410:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1c00 x1853163904605312/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767319175.991473:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904605184:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767319175.991562:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904605312:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767319176.119990:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319176.313611:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319177.145866:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319177.335305:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319178.167040:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319178.358806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319179.190970:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319179.382791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319180.225313:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319180.407100:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319181.110884:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319181.110893:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319181.110896:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319181.110899:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319181.240971:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319181.430832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319182.263425:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319182.454831:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319183.286930:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319183.478806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319184.310914:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319184.502851:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319185.335359:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319185.527087:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319186.230827:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319186.230838:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319186.230857:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319186.230860:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319186.358974:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319186.550873:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319187.383103:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319187.574765:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319188.406934:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319188.598910:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319189.433040:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319189.623055:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319190.457081:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319190.646871:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319191.353186:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319191.353200:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319191.353203:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319191.353205:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319191.479126:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319191.671589:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319192.503306:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319192.695608:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319193.528534:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319193.718799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319194.551116:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319194.743833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319195.574955:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319195.766878:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319196.470792:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319196.470805:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319196.470808:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319196.470812:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319196.604978:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319196.790817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319197.624460:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319197.814880:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319198.646901:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319198.844427:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319199.670894:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319199.864716:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767319200.566808:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319175/real 1767319175] req@ffff9d1f918d0700 x1853163904605184/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319200.566841:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d0700 x1853163904605184/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319200 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319200.566852:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319175/real 1767319175] req@ffff9d1f918d1c00 x1853163904605312/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319200.566864:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1c00 x1853163904605312/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319200 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767319200.566883:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319200.566917:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319200.566921:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904605184:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319200.566940:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319200.566971:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319200.566976:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904605312:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767319200.566995:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319200.567003:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767319200.567006:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319200.567012:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4423 00000100:00080000:3.0:1767319200.567021:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319200.567028:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767319200.567057:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71500 x1853163904605440/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319200.567085:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319200.567088:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767319200.567090:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319200.567094:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4423 00000100:00080000:3.0:1767319200.567097:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319200.567101:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767319200.567110:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767319200.567120:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70000 x1853163904605568/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767319200.567333:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904605440:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767319200.567433:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904605568:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767319200.695188:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319200.887610:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319201.718900:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319201.910872:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319202.742895:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319202.936098:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319203.766889:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319203.959449:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319204.792304:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319204.983121:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319205.691032:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319205.691045:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319205.691049:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319205.691075:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319205.814983:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319206.006798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319206.839060:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319207.030839:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319207.863569:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319208.054912:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319208.887453:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319209.079015:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319209.913458:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319210.103033:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319210.806805:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319210.806816:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319210.806819:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319210.806822:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319210.934924:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319211.126797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319211.958916:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319212.152230:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319212.983199:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319213.175030:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319214.008430:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319214.198828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319215.030937:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319215.222919:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319215.926923:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319215.926933:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319215.926936:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319215.926939:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319216.058719:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319216.246814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319217.079354:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319217.270873:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319218.103057:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319218.294950:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319219.126900:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319219.318841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319220.150914:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319220.342831:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319221.047038:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319221.047049:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319221.047052:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319221.047055:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319221.174942:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319221.366773:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319222.199389:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319222.390853:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319223.224429:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319223.414874:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319224.247242:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319224.438778:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767319225.146250:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319200/real 1767319200] req@ffff9d1f92f71500 x1853163904605440/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319225 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319225.146286:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71500 x1853163904605440/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319225 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319225.146298:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319200/real 1767319200] req@ffff9d1f92f70000 x1853163904605568/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319225 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319225.146309:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70000 x1853163904605568/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319225 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767319225.146328:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319225.146432:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319225.146438:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904605440:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319225.146445:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767319225.149545:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319225.149556:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767319225.149561:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319225.149571:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4448 00000100:00080000:3.0:1767319225.149583:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319225.149590:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767319225.149630:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73b80 x1853163904605696/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319225.149650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319225.149654:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767319225.149656:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319225.149659:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4448 00000100:00080000:3.0:1767319225.149662:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319225.149666:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767319225.149675:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767319225.150431:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71880 x1853163904605824/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319225.150752:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319225.150774:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904605568:192.168.202.140@tcp:38:kworker.0 00000100:00100000:0.0:1767319225.150809:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904605696:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767319225.151491:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904605824:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767319225.270897:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319225.462821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319226.294982:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319226.486806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319227.322167:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319227.510808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319228.343045:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319228.534838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319229.368958:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319229.558801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319230.263117:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319230.263127:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319230.263131:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319230.263134:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319230.391710:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319230.582934:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319231.415085:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319231.606804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319232.439838:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319232.630869:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319233.462985:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319233.654812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319234.487056:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319234.678819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319235.393509:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319235.393520:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319235.393535:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319235.393566:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319235.512023:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319235.702880:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319236.535188:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319236.726815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319237.559174:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319237.751819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319238.584225:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319238.774938:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319239.607139:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319239.798828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319240.503954:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319240.503966:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319240.503969:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319240.503972:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319240.630935:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319240.822813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319241.658663:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319241.846918:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319242.681016:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319242.871128:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319243.703036:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319243.894885:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319244.726909:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319244.920436:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319245.623016:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319245.623026:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319245.623029:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319245.623032:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319245.751329:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319245.942802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319246.774975:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319246.967718:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319247.798995:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319247.990785:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319248.823287:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319249.014825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319249.846915:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319250.044777:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319250.743018:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319250.743030:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319250.743032:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319250.743036:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767319250.743163:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319225/real 1767319225] req@ffff9d1f92f73b80 x1853163904605696/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319250 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319250.743197:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73b80 x1853163904605696/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319250 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319250.743216:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319225/real 1767319225] req@ffff9d1f92f71880 x1853163904605824/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319250 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319250.743229:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71880 x1853163904605824/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319250 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767319250.743248:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319250.743264:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319250.743269:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904605696:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767319250.743275:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319250.743293:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319250.743297:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904605824:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767319250.743354:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319250.743358:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767319250.743361:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319250.743366:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4472 00000100:00080000:3.0:1767319250.743382:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319250.743389:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767319250.743419:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71880 x1853163904605952/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319250.743452:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319250.743455:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767319250.743457:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319250.743461:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4472 00000100:00080000:3.0:1767319250.743472:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319250.743477:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767319250.743486:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767319250.743497:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73b80 x1853163904606080/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319250.743616:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904605952:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767319250.743728:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904606080:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767319250.870949:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319251.064693:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319251.894920:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319252.088918:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319252.920591:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319253.110795:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319253.946956:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319254.136953:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319254.969852:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319255.158925:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319255.863788:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319255.863891:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319255.864014:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319255.864019:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319255.991168:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319256.182882:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319257.015071:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319257.206879:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319258.039395:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319258.231532:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319259.062988:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319259.254907:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319260.087410:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319260.280619:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319260.983259:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319260.983270:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319260.983273:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319260.983295:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319261.111791:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319261.302817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319262.135735:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319262.326844:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319263.160913:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319263.352342:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319264.182941:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319264.375420:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319265.206899:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319265.399937:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319266.105462:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319266.105474:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319266.105477:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319266.105481:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319266.231062:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319266.423115:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319267.255371:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319267.448730:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319268.279072:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319268.472073:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319269.302887:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319269.495495:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319270.326959:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319270.518996:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319271.222799:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319271.222810:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319271.222813:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319271.222816:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319271.350900:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319271.542787:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319272.377007:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319272.567577:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319273.398911:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319273.590808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319274.425837:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319274.615508:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:0.0:1767319275.318864:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319250/real 1767319250] req@ffff9d1f92f71880 x1853163904605952/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319275.318915:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71880 x1853163904605952/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319275 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319275.318928:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319250/real 1767319250] req@ffff9d1f92f73b80 x1853163904606080/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319275.318942:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73b80 x1853163904606080/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319275 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767319275.319113:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319275.319218:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319275.319224:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904605952:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319275.319232:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767319275.326288:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319275.326303:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767319275.326309:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319275.326332:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4498 00000100:00080000:3.0:1767319275.326347:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319275.326356:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767319275.326422:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70000 x1853163904606208/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319275.326445:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319275.326449:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767319275.326451:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319275.326454:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4498 00000100:00080000:3.0:1767319275.326458:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319275.326471:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767319275.326482:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767319275.326515:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71500 x1853163904606336/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319275.334264:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319275.334282:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904606080:192.168.202.140@tcp:38:kworker.0 00000100:00100000:0.0:1767319275.334356:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904606208:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767319275.334533:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904606336:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767319275.449892:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319275.638856:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319276.470912:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319276.666890:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319277.495998:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319277.686866:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319278.519149:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319278.711195:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319279.542907:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319279.734849:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319280.438810:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319280.438822:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319280.438825:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319280.438828:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319280.566981:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319280.758778:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319281.594535:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319281.783233:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319282.615047:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319282.809247:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319283.639282:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319283.830776:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319284.663588:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319284.854880:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319285.558897:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319285.558908:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319285.558911:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319285.558914:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319285.687193:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319285.878772:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319286.714149:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319286.902827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319287.735870:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319287.926808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319288.758973:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319288.950831:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319289.783349:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319289.974801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319290.679971:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319290.679984:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319290.679987:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319290.679991:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319290.806986:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319290.998804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319291.831203:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319292.022821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319292.855320:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319293.046900:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319293.881977:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319294.071110:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319294.902921:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319295.094885:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319295.800375:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319295.800390:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319295.800393:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319295.800396:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319295.927193:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319296.120639:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319296.950904:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319297.142800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319297.974904:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319298.166804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319299.005407:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319299.190843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319300.023960:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319300.214832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319300.920714:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319300.920725:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319300.920728:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319300.920731:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1767319300.920790:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319275/real 1767319275] req@ffff9d1f92f70000 x1853163904606208/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319300 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319300.920817:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70000 x1853163904606208/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319300 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319300.920826:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319275/real 1767319275] req@ffff9d1f92f71500 x1853163904606336/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319300 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319300.920836:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71500 x1853163904606336/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319300 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767319300.920853:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319300.920874:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319300.920878:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904606208:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319300.920884:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319300.920889:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:0.0:1767319300.920892:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904606336:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767319300.920970:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319300.920975:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767319300.920978:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319300.920984:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4522 00000100:00080000:2.0:1767319300.920993:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319300.921000:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767319300.921042:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967ced80 x1853163904606464/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767319300.921066:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319300.921070:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767319300.921071:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319300.921075:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4522 00000100:00080000:2.0:1767319300.921078:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319300.921081:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767319300.921090:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767319300.921101:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cca80 x1853163904606592/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767319300.921159:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904606464:192.168.202.140@tcp:250:kworker.0 00000100:00100000:0.0:1767319300.921593:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904606592:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767319301.051822:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319301.238806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319302.070963:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319302.262830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319303.094903:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319303.286798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319304.118977:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319304.310920:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319305.144039:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319305.336005:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319306.040286:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319306.040297:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319306.040300:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319306.040304:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319306.167769:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319306.358919:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319307.191253:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319307.382834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319308.215708:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319308.406800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319309.239098:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319309.430826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319310.264515:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319310.454801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319311.161314:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319311.161325:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319311.161327:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319311.161331:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319311.287370:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319311.479172:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319312.312500:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319312.502811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319313.335449:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319313.527728:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319314.363096:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319314.551017:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319315.383086:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319315.575152:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319316.278838:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319316.278846:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319316.278849:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319316.278851:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319316.406920:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319316.598886:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319317.431204:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319317.624722:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319318.456899:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319318.647611:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319319.479202:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319319.671077:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319320.502954:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319320.694872:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319321.399010:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319321.399024:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319321.399027:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319321.399030:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319321.527185:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319321.718850:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319322.550942:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319322.742804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319323.575171:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319323.766834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319324.601290:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319324.790799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767319325.496299:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319300/real 1767319300] req@ffff9d1f967ced80 x1853163904606464/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319325 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319325.496349:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967ced80 x1853163904606464/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319325 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319325.496363:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319300/real 1767319300] req@ffff9d1f967cca80 x1853163904606592/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319325 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319325.496376:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cca80 x1853163904606592/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319325 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319325.496398:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319325.496457:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319325.496462:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904606464:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767319325.496470:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319325.496476:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319325.496479:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904606592:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767319325.496512:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319325.496519:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767319325.496530:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319325.496536:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4548 00000100:00080000:2.0:1767319325.496546:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319325.496555:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767319325.496603:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cca80 x1853163904606720/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319325.496700:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904606720:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767319325.496943:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319325.496948:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767319325.496950:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319325.496955:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4548 00000100:00080000:2.0:1767319325.496961:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319325.496966:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767319325.496978:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767319325.496993:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967ced80 x1853163904606848/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319325.497507:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904606848:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767319325.624891:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319325.814840:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319326.649279:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319326.838815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319327.676398:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319327.862836:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319328.695075:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319328.886809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319329.719422:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319329.910850:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319330.616261:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319330.616271:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319330.616273:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319330.616276:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319330.743005:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319330.934835:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319331.767348:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319331.958827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319332.794001:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319332.982893:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319333.815049:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319334.008300:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319334.840710:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319335.030940:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319335.734894:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319335.734906:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319335.734909:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319335.734912:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319335.863923:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319336.055014:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319336.889575:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319337.078816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319337.911372:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319338.106476:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319338.938116:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319339.126865:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319339.959836:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319340.151471:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319340.854948:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319340.854958:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319340.854961:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319340.854963:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319340.983723:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319341.174816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319342.007062:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319342.198893:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319343.031167:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319343.223824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319344.054914:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319344.248656:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319345.082657:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319345.270813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319345.975276:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319345.975286:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319345.975289:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319345.975292:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319346.102907:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319346.296641:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319347.127540:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319347.319318:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319348.156466:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319348.342911:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319349.175119:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319349.367268:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767319350.073076:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319325/real 1767319325] req@ffff9d1f967ced80 x1853163904606848/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319350 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319350.073115:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967ced80 x1853163904606848/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319350 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319350.073127:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319325/real 1767319325] req@ffff9d1f967cca80 x1853163904606720/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319350 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319350.073138:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cca80 x1853163904606720/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319350 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319350.073160:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319350.082132:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319350.082169:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904606848:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767319350.082183:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319350.082195:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319350.082199:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904606720:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767319350.082607:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319350.082619:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767319350.082624:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319350.082634:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4573 00000100:00080000:3.0:1767319350.082650:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319350.082658:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767319350.082892:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72a00 x1853163904606976/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319350.082990:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904606976:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767319350.083246:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319350.083252:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767319350.083254:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319350.083260:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4573 00000100:00080000:3.0:1767319350.083266:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319350.083271:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767319350.083299:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767319350.083316:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72d80 x1853163904607104/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319350.087464:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904607104:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767319350.204849:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319350.392880:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319351.223213:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319351.414859:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319352.248255:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319352.439404:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319353.272252:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319353.462959:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319354.295022:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319354.486974:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319355.190899:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319355.190913:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319355.190917:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319355.190922:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319355.322631:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319355.510868:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319356.343020:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319356.535209:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319357.367229:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319357.558865:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319358.391520:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319358.583897:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319359.418715:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319359.606946:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319360.310808:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319360.310819:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319360.310822:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319360.310825:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319360.440657:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319360.631989:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319361.466141:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319361.657284:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319362.488656:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319362.678866:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319363.511183:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319363.704450:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319364.535190:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319364.731746:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319365.430827:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319365.430839:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319365.430843:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319365.430848:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319365.559271:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319365.750876:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319366.582951:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319366.775799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319367.609775:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319367.798877:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319368.631308:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319368.823749:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319369.655120:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319369.848349:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319370.553770:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319370.553782:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319370.553785:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319370.553789:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319370.679002:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319370.873082:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319371.702950:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319371.894819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319372.726937:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319372.919283:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319373.751211:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319373.942821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319374.774939:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319374.966799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767319375.670897:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319350/real 1767319350] req@ffff9d1f92f72d80 x1853163904607104/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319375 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319375.670924:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72d80 x1853163904607104/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319375 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319375.670931:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319350/real 1767319350] req@ffff9d1f92f72a00 x1853163904606976/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319375 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319375.670939:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72a00 x1853163904606976/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319375 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319375.670951:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319375.670958:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319375.670960:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904607104:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767319375.670979:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319375.670982:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319375.670985:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904606976:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319375.671916:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319375.671924:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319375.671928:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319375.671946:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4597 00000100:00080000:0.0:1767319375.671956:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319375.671983:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767319375.672016:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033480 x1853163904607232/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319375.672045:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319375.672048:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319375.672050:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319375.672053:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4597 00000100:00080000:0.0:1767319375.672066:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319375.672070:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767319375.672077:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319375.672089:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030700 x1853163904607360/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319375.672162:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904607232:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767319375.672261:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904607360:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767319375.804999:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319375.990818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319376.823265:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319377.015444:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319377.846883:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319378.038805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319378.874606:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319379.063483:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319379.895265:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319380.086811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319380.791130:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319380.791141:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319380.791144:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319380.791148:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319380.919021:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319381.112532:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319381.942925:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319382.134832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319382.966985:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319383.159341:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319383.992172:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319384.183038:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319385.016063:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319385.207364:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319385.911146:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319385.911159:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319385.911162:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319385.911165:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319386.041373:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319386.231079:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319387.063081:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319387.254926:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319388.090699:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319388.280269:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319389.112509:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319389.303553:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319390.135350:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319390.326810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319391.031708:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319391.031719:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319391.031722:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319391.031725:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319391.159205:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319391.350892:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319392.185970:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319392.374802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319393.207564:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319393.398749:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319394.236282:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319394.422814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319395.257918:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319395.446851:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319396.151465:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319396.151481:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319396.151484:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319396.151487:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319396.286623:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319396.471471:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319397.303360:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319397.495191:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319398.328233:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319398.522221:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319399.350806:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319399.542852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767319400.247223:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319375/real 1767319375] req@ffff9d1f91033480 x1853163904607232/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319400 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319400.247256:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033480 x1853163904607232/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319400 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319400.247267:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319375/real 1767319375] req@ffff9d1f91030700 x1853163904607360/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319400 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319400.247278:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030700 x1853163904607360/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319400 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319400.247295:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319400.247326:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319400.247330:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904607232:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767319400.247336:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319400.247340:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319400.247344:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904607360:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767319400.247368:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319400.247376:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767319400.247430:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319400.247627:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4623 00000100:00080000:3.0:1767319400.247636:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319400.247644:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767319400.247675:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72680 x1853163904607488/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319400.247734:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319400.247738:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767319400.247740:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319400.247751:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4623 00000100:00080000:3.0:1767319400.247755:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319400.247759:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767319400.247768:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767319400.247780:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71f80 x1853163904607616/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319400.248074:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904607488:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767319400.248176:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904607616:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767319400.378055:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319400.567218:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319401.399231:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319401.591211:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319402.426204:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319402.614806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319403.447251:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319403.641321:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319404.477114:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319404.663000:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319405.367406:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319405.367417:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319405.367420:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319405.367422:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319405.494936:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319405.686825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319406.528997:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319406.710830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319407.544092:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319407.734892:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319408.572814:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319408.759648:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319409.591380:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319409.782980:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319410.487044:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319410.487055:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319410.487057:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319410.487060:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319410.615894:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319410.809459:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319411.639917:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319411.831445:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319412.663044:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319412.855382:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319413.686953:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319413.879742:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319414.711173:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319414.902815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319415.606808:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319415.606818:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319415.606820:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319415.606822:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319415.735579:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319415.926790:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319416.760293:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319416.950959:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319417.783003:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319417.974920:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319418.810269:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319418.998776:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319419.832203:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319420.022821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319420.726964:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319420.726976:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319420.726980:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319420.726984:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319420.857099:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319421.046822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319421.880873:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319422.071791:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319422.905114:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319423.095419:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319423.927094:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319424.118874:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319424.951202:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319425.143009:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767319425.847098:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319400/real 1767319400] req@ffff9d1f92f72680 x1853163904607488/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767319425.847113:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319425.847121:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319425.847125:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319425.847128:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1767319425.847147:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72680 x1853163904607488/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319425 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319425.847159:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319400/real 1767319400] req@ffff9d1f92f71f80 x1853163904607616/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319425.847172:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71f80 x1853163904607616/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319425 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319425.847193:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319425.847254:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319425.847259:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904607488:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767319425.847267:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319425.847272:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319425.847276:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904607616:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767319425.847437:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319425.847441:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767319425.847453:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319425.847460:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4647 00000100:00080000:3.0:1767319425.847470:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319425.847477:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767319425.847511:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73480 x1853163904607744/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319425.847543:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319425.847546:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767319425.847548:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319425.847551:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4647 00000100:00080000:3.0:1767319425.847554:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319425.847558:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767319425.847567:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767319425.847577:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71180 x1853163904607872/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319425.848305:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904607744:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767319425.848423:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904607872:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767319425.975309:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319426.166912:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319426.999119:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319427.190998:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319428.023002:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319428.216572:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319429.046922:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319429.238957:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319430.073226:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319430.262879:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319430.966810:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319430.966833:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319430.966837:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319430.966863:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319431.094998:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319431.286868:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319432.119101:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319432.310861:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319433.142942:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319433.334948:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319434.167706:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319434.358855:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319435.191245:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319435.382884:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319436.086935:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319436.086947:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319436.086951:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319436.086954:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319436.219366:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319436.406828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319437.240155:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319437.430933:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319438.263339:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319438.454832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319439.288456:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319439.479822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319440.312642:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319440.502914:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319441.207143:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319441.207157:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319441.207161:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319441.207164:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319441.335777:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319441.528929:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319442.368199:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319442.550841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319443.383211:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319443.575893:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319444.407786:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319444.599507:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319445.431524:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319445.622812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319446.326895:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319446.326909:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319446.326912:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319446.326916:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319446.454936:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319446.646981:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319447.478994:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319447.670782:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319448.508112:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319448.695958:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319449.527025:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319449.719617:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767319450.422948:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319425/real 1767319425] req@ffff9d1f92f73480 x1853163904607744/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319450 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319450.422984:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73480 x1853163904607744/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319450 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319450.422993:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319425/real 1767319425] req@ffff9d1f92f71180 x1853163904607872/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319450 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319450.423005:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71180 x1853163904607872/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319450 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319450.423024:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319450.423060:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319450.423084:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904607744:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767319450.423090:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319450.423097:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319450.423110:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904607872:192.168.202.140@tcp:38:kworker.0 00000100:00080000:0.0:1767319450.423203:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319450.423208:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319450.423212:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319450.423217:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4673 00000100:00080000:0.0:1767319450.423225:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319450.423231:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767319450.423260:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91033100 x1853163904608000/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319450.423285:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319450.423288:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319450.423297:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319450.423301:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4673 00000100:00080000:0.0:1767319450.423306:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319450.423310:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767319450.423319:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319450.423333:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030380 x1853163904608128/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319450.423399:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904608000:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767319450.423489:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904608128:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767319450.550911:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319450.743050:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319451.574960:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319451.766891:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319452.598927:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319452.790882:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319453.623015:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319453.816472:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319454.646984:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319454.838807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319455.543525:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319455.543536:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319455.543538:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319455.543541:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319455.670897:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319455.863626:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319456.698919:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319456.887658:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319457.724699:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319457.911014:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319458.742903:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319458.934806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319459.767278:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319459.958803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319460.662817:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319460.662828:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319460.662831:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319460.662834:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319460.790988:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319460.982808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319461.817572:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319462.006865:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319462.839249:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319463.030827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319463.862908:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319464.056633:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319464.894891:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319465.080103:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319465.783285:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319465.783296:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319465.783299:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319465.783303:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319465.910912:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319466.102807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319466.935365:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319467.126918:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319467.959618:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319468.150813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319468.985168:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319469.176271:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319470.007011:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319470.200001:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319470.903078:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319470.903091:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319470.903093:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319470.903096:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319471.033053:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319471.222822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319472.063840:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319472.247194:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319473.078994:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319473.270838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319474.102962:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319474.295479:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319475.127794:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319475.319054:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319476.022790:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319476.022802:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1767319476.022803:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319450/real 1767319450] req@ffff9d1f91033100 x1853163904608000/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319475 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767319476.022805:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319476.022812:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1767319476.022867:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91033100 x1853163904608000/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319475 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319476.022877:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319450/real 1767319450] req@ffff9d1f91030380 x1853163904608128/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319475 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319476.022888:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030380 x1853163904608128/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319475 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319476.022918:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319476.022940:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319476.022944:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91033100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904608000:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767319476.022951:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319476.022957:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319476.022960:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904608128:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767319476.022967:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319476.022972:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767319476.022975:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319476.022982:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4698 00000100:00080000:3.0:1767319476.022990:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319476.022997:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767319476.023038:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73800 x1853163904608256/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319476.023122:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904608256:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767319476.023224:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319476.023227:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767319476.023228:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319476.023231:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4698 00000100:00080000:3.0:1767319476.023234:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319476.023238:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767319476.023246:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767319476.023255:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70700 x1853163904608384/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319476.023575:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904608384:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767319476.151097:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319476.342990:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319477.175761:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319477.366863:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319478.198901:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319478.390811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319479.222906:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319479.414882:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319480.248952:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319480.441500:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319481.142830:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319481.142846:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319481.142851:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319481.142854:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319481.271283:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319481.462825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319482.294895:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319482.487199:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319483.319373:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319483.510807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319484.343559:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319484.534912:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319485.367100:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319485.559110:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319486.265231:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319486.265242:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319486.265245:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319486.265249:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319486.392314:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319486.583046:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319487.414924:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319487.608422:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319488.438865:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319488.631225:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319489.462950:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319489.654781:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319490.489170:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319490.681644:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319491.388221:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319491.388235:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319491.390516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319491.390524:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319491.512670:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319491.703214:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319492.535207:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319492.730558:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319493.570998:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319493.756810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319494.583856:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319494.774883:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319495.606911:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319495.798994:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319496.503017:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319496.503026:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319496.503028:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319496.503048:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319496.631057:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319496.823431:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319497.655842:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319497.846824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319498.678916:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319498.870837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319499.702983:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319499.894863:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319500.731138:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319500.918957:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319501.622932:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319501.622949:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319501.622953:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319501.622956:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1767319501.623061:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319476/real 1767319476] req@ffff9d1f92f70700 x1853163904608384/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319501.623100:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70700 x1853163904608384/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319501 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319501.623112:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319476/real 1767319476] req@ffff9d1f92f73800 x1853163904608256/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319501.623152:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73800 x1853163904608256/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319501 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767319501.623173:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319501.623203:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319501.623209:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904608384:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767319501.623215:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319501.623220:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319501.623223:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904608256:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319501.623299:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319501.623303:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319501.623308:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319501.623314:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4723 00000100:00080000:0.0:1767319501.623324:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319501.623331:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767319501.623370:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032d80 x1853163904608512/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319501.623394:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319501.623398:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319501.623400:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319501.623403:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4723 00000100:00080000:0.0:1767319501.623407:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319501.623410:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767319501.623421:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319501.623432:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031880 x1853163904608640/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319501.623656:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904608512:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767319501.623852:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904608640:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767319501.751246:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319501.942895:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319502.776215:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319502.966871:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319503.800261:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319503.991538:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319504.823487:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319505.018550:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319505.849256:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319506.038865:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319506.744006:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319506.744029:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319506.744033:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319506.744036:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319506.875238:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319507.062841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319507.896908:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319508.086877:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319508.919386:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319509.114104:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319509.949762:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319510.135566:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319510.967836:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319511.159228:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319511.866441:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319511.866456:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319511.866459:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319511.866463:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319511.991570:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319512.183434:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319513.015017:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319513.206854:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319514.042633:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319514.230808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319515.065945:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319515.254860:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319516.087421:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319516.278850:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319516.983157:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319516.983172:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319516.983176:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319516.983180:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319517.111105:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319517.306027:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319518.135239:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319518.326856:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319519.159434:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319519.350926:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319520.183607:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319520.376564:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319521.207528:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319521.399035:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319522.103278:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319522.103288:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319522.103290:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319522.103293:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319522.231119:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319522.422875:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319523.255029:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319523.447621:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319524.280512:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319524.471874:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319525.303371:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319525.495505:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767319526.199068:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319501/real 1767319501] req@ffff9d1f91032d80 x1853163904608512/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319526.199114:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032d80 x1853163904608512/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319526 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319526.199126:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319501/real 1767319501] req@ffff9d1f91031880 x1853163904608640/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319526.199138:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031880 x1853163904608640/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319526 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767319526.199173:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319526.199208:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319526.199213:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904608512:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767319526.199220:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319526.199226:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319526.199229:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904608640:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767319526.199288:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319526.199296:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767319526.199299:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319526.199307:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4749 00000100:00080000:3.0:1767319526.199319:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319526.199328:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767319526.199373:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70380 x1853163904608768/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319526.199406:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319526.199410:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767319526.199412:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319526.199416:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4749 00000100:00080000:3.0:1767319526.199421:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319526.199448:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767319526.199461:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767319526.199475:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70e00 x1853163904608896/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319526.200198:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904608768:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767319526.200422:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904608896:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767319526.327438:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319526.519496:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319527.350936:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319527.543414:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319528.375555:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319528.566856:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319529.400262:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319529.592451:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319530.423355:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319530.617345:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319531.320239:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319531.320256:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319531.320260:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319531.320264:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319531.447881:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319531.638868:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319532.473492:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319532.665051:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319533.495792:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319533.686913:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319534.528050:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319534.711202:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319535.543451:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319535.741243:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319536.441493:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319536.441505:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319536.441508:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319536.441511:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319536.568135:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319536.762211:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319537.603459:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319537.782822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319538.615267:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319538.809148:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319539.642428:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319539.831607:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319540.663698:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319540.855049:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319541.562969:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319541.562978:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319541.562981:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319541.562983:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319541.687776:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319541.879003:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319542.711085:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319542.904263:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319543.736653:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319543.927913:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319544.759126:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319544.951914:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319545.783611:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319545.978015:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319546.678811:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319546.678822:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319546.678825:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319546.678828:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319546.810450:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319546.999006:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319547.830965:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319548.023735:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319548.856073:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319549.046856:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319549.879011:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319550.071095:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319550.902993:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319551.094825:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767319551.798869:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319526/real 1767319526] req@ffff9d1f92f70380 x1853163904608768/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319551.798904:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70380 x1853163904608768/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319551 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319551.798916:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319526/real 1767319526] req@ffff9d1f92f70e00 x1853163904608896/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767319551.798923:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1767319551.798930:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70e00 x1853163904608896/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319551 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767319551.798931:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319551.798937:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319551.798946:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319551.798958:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319551.798971:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319551.798976:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904608768:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319551.798980:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319551.798985:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319551.799066:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319551.802905:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4773 00000100:00080000:2.0:1767319551.802905:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319551.802936:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319551.802937:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00080000:0.0:1767319551.802947:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767319551.802947:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904608896:192.168.202.140@tcp:38:kworker.0 00000100:00100000:0.0:1767319551.803163:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031180 x1853163904609024/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319551.803229:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904609024:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319551.803304:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319551.803309:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319551.803312:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319551.803316:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4773 00000100:00080000:0.0:1767319551.803321:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319551.803325:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767319551.803334:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319551.803346:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030000 x1853163904609152/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319551.803513:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904609152:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767319551.927218:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319552.118851:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319552.950948:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319553.142841:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319553.977887:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319554.167282:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319554.999149:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319555.190821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319556.023206:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319556.214806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319556.921904:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319556.921913:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319556.921916:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319556.921920:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319557.046908:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319557.238795:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319558.070938:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319558.262789:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319559.094923:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319559.286785:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319560.119008:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319560.310955:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319561.147757:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319561.334822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319562.038825:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319562.038837:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319562.038840:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319562.038843:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319562.166975:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319562.359138:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319563.190883:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319563.382897:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319564.214979:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319564.406826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319565.239071:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319565.430957:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319566.262954:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319566.454896:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319567.159094:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319567.159105:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319567.159108:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319567.159111:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319567.286975:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319567.478976:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319568.311521:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319568.502846:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319569.335076:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319569.526799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319570.365963:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319570.551329:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319571.386125:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319571.574884:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319572.281443:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319572.281457:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319572.281460:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319572.281464:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319572.410215:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319572.600002:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319573.430959:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319573.627718:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319574.457887:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319574.646856:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319575.479383:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319575.670843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767319576.374955:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319551/real 1767319551] req@ffff9d1f91030000 x1853163904609152/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319576 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319576.374996:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030000 x1853163904609152/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319576 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319576.375008:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319551/real 1767319551] req@ffff9d1f91031180 x1853163904609024/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319576 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319576.375021:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031180 x1853163904609024/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319576 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319576.375040:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319576.375050:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319576.375073:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904609152:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767319576.375079:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319576.375085:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319576.375088:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904609024:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319576.375295:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319576.375301:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319576.375304:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319576.375309:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4799 00000100:00080000:0.0:1767319576.375318:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319576.375325:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767319576.375356:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032300 x1853163904609280/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319576.375389:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319576.375402:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319576.375404:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319576.375408:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4799 00000100:00080000:0.0:1767319576.375412:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319576.375416:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767319576.375425:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319576.375437:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030a80 x1853163904609408/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319576.375823:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904609280:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767319576.375951:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904609408:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767319576.502902:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319576.695266:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319577.526899:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319577.719778:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319578.550933:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319578.744950:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319579.579506:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319579.767041:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319580.599861:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319580.791049:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319581.499622:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319581.499632:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319581.499635:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319581.499638:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319581.625101:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319581.814818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319582.649158:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319582.838818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319583.671445:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319583.865161:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319584.695790:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319584.886807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319585.721582:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319585.910800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319586.615766:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319586.615778:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319586.615781:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319586.615785:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319586.743281:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319586.934829:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319587.772635:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319587.960708:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319588.791658:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319588.983537:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319589.815077:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319590.007389:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319590.839626:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319591.031461:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319591.735090:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319591.735101:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319591.735104:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319591.735139:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319591.863080:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319592.055458:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319592.887464:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319593.078811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319593.911863:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319594.106111:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319594.935566:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319595.126982:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319595.959138:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319596.150824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319596.856069:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319596.856082:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319596.856084:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319596.856088:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319596.982919:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319597.174792:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319598.007894:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319598.198823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319599.030954:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319599.223870:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319600.054914:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319600.249652:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319601.079262:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319601.270800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767319601.974818:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319576/real 1767319576] req@ffff9d1f91032300 x1853163904609280/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319601 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319601.974853:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032300 x1853163904609280/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319601 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319601.974864:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319576/real 1767319576] req@ffff9d1f91030a80 x1853163904609408/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319601 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319601.974877:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030a80 x1853163904609408/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319601 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767319601.974903:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319601.974912:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319601.974917:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319601.974920:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319601.974929:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319601.974953:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319601.974957:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904609280:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767319601.974963:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319601.974967:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319601.974970:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904609408:192.168.202.140@tcp:38:kworker.0 00000100:00080000:3.0:1767319601.975079:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319601.975084:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767319601.975087:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319601.975094:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4824 00000100:00080000:3.0:1767319601.975103:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319601.975110:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767319601.975142:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f72300 x1853163904609536/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319601.975205:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904609536:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767319601.975400:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319601.975404:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767319601.975406:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319601.975411:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4824 00000100:00080000:3.0:1767319601.975415:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319601.975427:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767319601.975437:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767319601.975451:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f73100 x1853163904609664/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319601.976005:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904609664:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767319602.103103:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319602.294997:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319603.127078:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319603.318856:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319604.152368:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319604.344036:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319605.177634:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319605.366774:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319606.201570:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319606.391064:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319607.094801:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319607.094811:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319607.094814:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319607.094816:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319607.224529:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319607.414822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319608.248443:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319608.438835:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319609.270994:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319609.462973:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319610.295568:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319610.486858:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319611.319645:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319611.510810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319612.214841:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319612.214854:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319612.214858:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319612.214883:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319612.343421:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319612.534803:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319613.367657:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319613.558799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319614.390917:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319614.582915:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319615.414927:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319615.606775:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319616.439902:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319616.630802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319617.334827:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319617.334839:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319617.334842:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319617.334845:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319617.464321:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319617.656595:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319618.486862:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319618.678908:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319619.512077:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319619.702932:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319620.538272:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319620.727495:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319621.558868:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319621.751509:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319622.454984:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319622.455004:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319622.455025:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319622.455029:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319622.583135:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319622.774808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319623.611327:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319623.798988:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319624.632658:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319624.827052:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319625.656271:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319625.847163:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:2.0:1767319626.550832:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319601/real 1767319601] req@ffff9d1f92f73100 x1853163904609664/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319626 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319626.550869:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f73100 x1853163904609664/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319626 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319626.550881:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319601/real 1767319601] req@ffff9d1f92f72300 x1853163904609536/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319626 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1767319626.550892:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f72300 x1853163904609536/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319626 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1767319626.550916:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319626.550951:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319626.550956:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f73100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904609664:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767319626.550981:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319626.550988:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:2.0:1767319626.550991:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f72300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904609536:192.168.202.140@tcp:250:kworker.0 00000100:00080000:0.0:1767319626.553136:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319626.553145:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319626.553150:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319626.553160:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4849 00000100:00080000:0.0:1767319626.553172:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319626.553180:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767319626.553219:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031500 x1853163904609792/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319626.553257:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319626.553261:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319626.553262:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319626.553266:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4849 00000100:00080000:0.0:1767319626.553269:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319626.553273:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767319626.553282:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319626.553293:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032680 x1853163904609920/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1767319626.555142:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904609792:192.168.202.140@tcp:250:kworker.0 00000100:00100000:2.0:1767319626.555309:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904609920:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767319626.679446:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319626.872216:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319627.702872:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319627.894871:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319628.726982:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319628.920392:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319629.751323:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319629.942854:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319630.775103:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319630.967392:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319631.672874:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319631.672886:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319631.672889:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319631.672906:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319631.799778:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319631.992126:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319632.822889:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319633.015120:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319633.847084:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319634.038859:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319634.876488:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319635.063719:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319635.895016:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319636.087108:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319636.797891:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319636.797904:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319636.797907:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319636.797911:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319636.920038:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319637.111105:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319637.942886:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319638.134800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319638.966865:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319639.158874:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319639.990983:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319640.182801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319641.015164:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319641.206820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319641.922625:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319641.922635:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319641.922639:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319641.922642:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319642.040072:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319642.230830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319643.065019:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319643.255275:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319644.089072:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319644.278808:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319645.112035:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319645.302779:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319646.134953:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319646.327225:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319647.031777:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319647.031786:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319647.031789:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319647.031793:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319647.158906:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319647.350817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319648.182939:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319648.374816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319649.207581:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319649.398798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319650.231039:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319650.426487:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767319651.127440:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319626/real 1767319626] req@ffff9d1f91031500 x1853163904609792/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319651.127474:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031500 x1853163904609792/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319651 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319651.127484:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319626/real 1767319626] req@ffff9d1f91032680 x1853163904609920/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319651.127495:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032680 x1853163904609920/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319651 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319651.127514:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319651.127536:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319651.127541:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904609792:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767319651.127643:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319651.127648:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767319651.127651:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319651.127657:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4874 00000100:00080000:3.0:1767319651.127666:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319651.127672:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767319651.127737:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71c00 x1853163904610048/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319651.127751:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319651.127754:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319651.128659:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319651.128699:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319651.128704:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904609920:192.168.202.140@tcp:38:kworker.0 00000100:00100000:1.0:1767319651.128735:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904610048:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767319651.128935:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319651.128940:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767319651.128942:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319651.128946:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4874 00000100:00080000:3.0:1767319651.128952:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319651.128957:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767319651.128966:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767319651.128985:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70a80 x1853163904610176/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319651.129102:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904610176:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767319651.255064:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319651.446890:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319652.278889:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319652.470835:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319653.302964:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319653.494838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319654.330066:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319654.519003:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319655.350867:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319655.546705:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319656.246964:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319656.246974:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319656.246977:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319656.246980:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319656.378262:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319656.566871:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319657.398919:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319657.590812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319658.423123:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319658.614886:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319659.446997:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319659.638953:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319660.475842:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319660.662809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319661.366849:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319661.366888:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319661.366891:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319661.366895:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319661.496095:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319661.686938:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319662.524239:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319662.710800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319663.543208:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319663.734848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319664.567010:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319664.758779:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319665.591011:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319665.783150:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319666.486874:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319666.486886:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319666.486889:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319666.486893:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319666.614991:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319666.806985:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319667.639034:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319667.830814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319668.663348:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319668.855170:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319669.688471:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319669.879231:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319670.711148:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319670.902818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319671.609824:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319671.609834:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319671.609837:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319671.609841:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319671.737610:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319671.928124:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319672.758843:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319672.950891:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319673.783146:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319673.974816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319674.807218:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319674.998807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319675.830901:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319676.022843:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319676.726950:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319676.726963:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319676.726966:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319676.726969:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1767319676.727060:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319651/real 1767319651] req@ffff9d1f92f70a80 x1853163904610176/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319676.727102:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70a80 x1853163904610176/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319676 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319676.727113:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319651/real 1767319651] req@ffff9d1f92f71c00 x1853163904610048/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319676.727125:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71c00 x1853163904610048/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319676 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319676.727143:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319676.727164:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319676.727168:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904610176:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767319676.727174:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319676.727179:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319676.727208:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904610048:192.168.202.140@tcp:250:kworker.0 00000100:00080000:2.0:1767319676.727427:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319676.727432:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767319676.727435:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319676.727440:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4898 00000100:00080000:2.0:1767319676.727449:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319676.727455:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767319676.727495:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cdc00 x1853163904610304/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767319676.727527:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319676.727530:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767319676.727531:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319676.727535:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4898 00000100:00080000:2.0:1767319676.727539:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319676.727543:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767319676.727698:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767319676.727713:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cf100 x1853163904610432/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319676.727777:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904610304:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767319676.728041:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904610432:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767319676.854916:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319677.047085:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319677.878979:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319678.070986:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319678.904104:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319679.094799:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319679.926927:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319680.119793:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319680.950949:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319681.142885:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319681.846809:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319681.846817:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319681.846820:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319681.846822:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319681.975103:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319682.166835:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319682.999239:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319683.190847:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319684.023447:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319684.215754:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319685.046905:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319685.238807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319686.071342:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319686.262867:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319686.966828:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319686.966838:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319686.966841:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319686.966844:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319687.094880:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319687.288204:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319688.118935:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319688.311001:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319689.143619:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319689.334828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319690.169725:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319690.360159:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319691.191918:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319691.382853:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319692.086913:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319692.086928:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319692.086931:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319692.086935:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319692.224639:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319692.406821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319693.241574:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319693.432479:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319694.266826:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319694.457056:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319695.287204:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319695.478848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319696.311737:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319696.502856:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319697.207590:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319697.207605:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319697.207609:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319697.207613:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319697.337726:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319697.526852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319698.359252:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319698.551125:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319699.385710:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319699.574812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319700.413535:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319700.598814:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767319701.302846:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319676/real 1767319676] req@ffff9d1f967cdc00 x1853163904610304/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319701 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319701.302888:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cdc00 x1853163904610304/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319701 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319701.302912:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319676/real 1767319676] req@ffff9d1f967cf100 x1853163904610432/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319701 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319701.302925:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cf100 x1853163904610432/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319701 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767319701.302945:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767319701.303032:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767319701.303038:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904610304:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767319701.303045:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319701.307291:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319701.307312:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319701.307316:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319701.307327:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4924 00000100:00080000:0.0:1767319701.307339:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319701.307346:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767319701.307396:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032680 x1853163904610560/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319701.307416:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319701.307434:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319701.307437:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319701.307441:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4924 00000100:00080000:0.0:1767319701.307445:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319701.307449:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767319701.307459:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319701.307472:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91031500 x1853163904610688/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319701.309803:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767319701.309816:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904610432:192.168.202.140@tcp:38:kworker.0 00000100:00100000:3.0:1767319701.309866:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904610560:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767319701.309986:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904610688:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767319701.431315:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319701.623373:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319702.463421:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319702.646810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319703.479750:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319703.670809:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319704.504797:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319704.696383:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319705.531776:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319705.718979:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319706.424293:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319706.424339:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319706.424344:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319706.424348:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319706.557548:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319706.742902:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319707.575106:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319707.766879:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319708.600242:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319708.791335:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319709.623982:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319709.815089:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319710.647212:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319710.839028:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319711.542916:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319711.542926:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319711.542929:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319711.542933:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319711.673826:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319711.862983:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319712.697724:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319712.887478:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319713.720761:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319713.910804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319714.742987:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319714.934960:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319715.766920:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319715.964521:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319716.662841:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319716.662852:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319716.662855:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319716.662859:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319716.790924:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319716.982889:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319717.815098:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319718.006815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319718.838886:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319719.031800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319719.862846:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319720.054785:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319720.887145:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319721.078821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319721.783126:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319721.783137:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319721.783140:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319721.783151:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319721.911011:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319722.103491:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319722.937718:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319723.126837:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319723.959468:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319724.151820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319724.984184:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319725.175736:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319726.009099:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319726.198782:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319726.903252:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319726.903263:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319726.903266:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319726.903269:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1767319726.906156:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319701/real 1767319701] req@ffff9d1f91032680 x1853163904610560/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319726 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319726.906193:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032680 x1853163904610560/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319726 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319726.906203:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319701/real 1767319701] req@ffff9d1f91031500 x1853163904610688/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319726 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319726.906214:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91031500 x1853163904610688/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319726 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319726.906234:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319726.906332:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319726.906337:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904610560:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767319726.906344:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319726.909281:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319726.909292:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767319726.909296:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319726.909333:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4948 00000100:00080000:2.0:1767319726.909344:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319726.909351:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767319726.909391:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967ce300 x1853163904610816/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767319726.909409:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319726.909412:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767319726.909414:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319726.909418:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4948 00000100:00080000:2.0:1767319726.909422:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319726.909425:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767319726.909435:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767319726.909448:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cdf80 x1853163904610944/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767319726.910377:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319726.910385:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91031500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904610688:192.168.202.140@tcp:38:kworker.0 00000100:00100000:1.0:1767319726.910426:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904610816:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767319726.910600:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904610944:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767319727.031009:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319727.222823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319728.056827:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319728.246815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319729.081380:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319729.270833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319730.103274:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319730.295076:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319731.127297:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319731.318804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319732.022997:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319732.023008:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319732.023010:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319732.023014:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319732.151813:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319732.342885:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319733.175744:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319733.367472:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319734.198999:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319734.392274:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319735.223028:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319735.414823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319736.247250:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319736.438818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319737.144337:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319737.144348:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319737.144351:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319737.144354:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319737.270944:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319737.462966:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319738.298329:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319738.486848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319739.319531:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319739.510805:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319740.343187:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319740.534835:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319741.368275:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319741.558821:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319742.263618:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319742.263648:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319742.263651:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319742.263654:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319742.399325:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319742.582812:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319743.415025:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319743.610348:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319744.439209:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319744.630807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319745.463971:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319745.654891:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319746.487122:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319746.679187:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319747.382901:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319747.382922:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319747.382926:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319747.382929:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319747.510968:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319747.702798:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319748.534903:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319748.727043:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319749.559933:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319749.750875:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319750.583796:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319750.774807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767319751.478822:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319726/real 1767319726] req@ffff9d1f967ce300 x1853163904610816/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319751.478858:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967ce300 x1853163904610816/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319751 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319751.478880:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319726/real 1767319726] req@ffff9d1f967cdf80 x1853163904610944/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319751.478894:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cdf80 x1853163904610944/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319751 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319751.478921:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319751.478952:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319751.478956:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904610816:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767319751.478963:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319751.478969:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319751.478972:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904610944:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767319751.479058:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319751.479063:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767319751.479066:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319751.479071:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4974 00000100:00080000:2.0:1767319751.479080:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319751.479102:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767319751.479132:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cd500 x1853163904611072/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767319751.479157:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319751.479160:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767319751.479162:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319751.479166:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4974 00000100:00080000:2.0:1767319751.479170:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319751.479173:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767319751.479182:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767319751.479193:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cea00 x1853163904611200/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319751.479985:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904611072:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767319751.480094:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904611200:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767319751.606860:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319751.799347:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319752.631208:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319752.824301:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319753.655322:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319753.848052:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319754.680465:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319754.870858:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319755.703455:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319755.894838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319756.599240:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319756.599251:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319756.599254:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319756.599257:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319756.726917:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319756.918816:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319757.760305:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319757.942779:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319758.775027:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319758.966800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319759.798909:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319759.990894:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319760.822949:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319761.014802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319761.718804:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319761.718831:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319761.718835:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319761.718856:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319761.849435:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319762.038819:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319762.871442:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319763.062866:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319763.894932:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319764.086884:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319764.918969:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319765.110901:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319765.943570:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319766.136005:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319766.839841:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319766.839853:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319766.839856:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319766.839859:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319766.967649:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319767.158944:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319767.992804:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319768.182966:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319769.015075:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319769.207327:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319770.039864:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319770.230866:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319771.063840:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319771.254871:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319771.959110:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319771.959133:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319771.959136:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319771.959140:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319772.095843:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319772.278807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319773.111584:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319773.303409:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319774.135057:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319774.326848:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319775.158997:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319775.350827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767319776.055424:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319751/real 1767319751] req@ffff9d1f967cd500 x1853163904611072/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319776.055767:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cd500 x1853163904611072/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319776 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319776.055781:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319751/real 1767319751] req@ffff9d1f967cea00 x1853163904611200/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319776.055795:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cea00 x1853163904611200/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319776 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767319776.055816:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767319776.055892:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767319776.055897:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904611072:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767319776.055905:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1767319776.058886:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319776.058902:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319776.058908:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319776.058923:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 4999 00000100:00080000:0.0:1767319776.058939:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319776.058948:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:0.0:1767319776.059016:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91030a80 x1853163904611328/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319776.059067:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319776.059072:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319776.059074:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319776.059079:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 4999 00000100:00080000:0.0:1767319776.059083:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319776.059088:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:0.0:1767319776.059100:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319776.059115:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f91032300 x1853163904611456/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319776.065000:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767319776.065016:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904611200:192.168.202.140@tcp:38:kworker.0 00000100:00100000:3.0:1767319776.065079:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904611328:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767319776.065232:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904611456:192.168.202.140@tcp:38:kworker.0 00010000:00010000:0.0:1767319776.183236:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319776.375232:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319777.207042:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319777.399316:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319778.230977:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319778.429045:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319779.254960:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319779.446863:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319780.279173:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319780.474186:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319781.175753:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319781.175768:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319781.175771:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319781.175774:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319781.303385:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319781.494840:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319782.327206:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319782.518832:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319783.351028:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319783.543104:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319784.376066:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319784.566852:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319785.401382:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319785.590948:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319786.294978:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319786.294994:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319786.294998:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319786.295002:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319786.424395:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319786.614854:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319787.447262:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319787.638849:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319788.470963:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319788.666775:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319789.496137:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319789.688248:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319790.519261:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319790.713833:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319791.419276:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319791.419290:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319791.419293:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319791.419297:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319791.543023:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319791.736517:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319792.567098:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319792.758772:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319793.590991:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319793.784994:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319794.615040:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319794.806823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319795.639847:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319795.830817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319796.535959:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319796.535973:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319796.535976:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319796.535979:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319796.666629:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319796.857041:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319797.687616:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319797.878826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319798.711019:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319798.902942:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319799.735588:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319799.927096:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319800.759064:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319800.950822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319801.654793:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319801.654804:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319801.654807:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319801.654811:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1767319801.658173:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319776/real 1767319776] req@ffff9d1f91030a80 x1853163904611328/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319801.658238:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91030a80 x1853163904611328/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319801 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319801.658249:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319776/real 1767319776] req@ffff9d1f91032300 x1853163904611456/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319801.658260:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f91032300 x1853163904611456/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319801 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319801.658281:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319801.658522:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319801.658528:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91030a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904611328:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767319801.658536:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767319801.665217:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319801.665228:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767319801.665233:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319801.665242:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 5023 00000100:00080000:3.0:1767319801.667233:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319801.667245:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:3.0:1767319801.667285:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f71c00 x1853163904611584/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319801.667643:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319801.667649:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767319801.667651:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319801.667657:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 5023 00000100:00080000:3.0:1767319801.667661:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319801.667665:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:3.0:1767319801.667675:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767319801.667711:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f92f70a80 x1853163904611712/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767319801.668526:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319801.668536:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f91032300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904611456:192.168.202.140@tcp:38:kworker.0 00000100:00100000:1.0:1767319801.668577:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904611584:192.168.202.140@tcp:250:kworker.0 00000100:00100000:1.0:1767319801.668696:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904611712:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767319801.783708:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319801.975417:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319802.807364:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319802.999489:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319803.833936:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319804.023012:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319804.854899:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319805.046901:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319805.878943:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319806.071671:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319806.774912:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319806.774923:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319806.774943:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319806.774945:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319806.906249:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319807.094838:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319807.927240:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319808.118811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319808.954458:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319809.144792:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319809.975071:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319810.167138:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319810.998883:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319811.190807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319811.895154:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319811.895164:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319811.895167:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319811.895171:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319812.026192:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319812.216513:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319813.046969:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319813.239240:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319814.074841:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319814.263131:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319815.094983:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319815.286810:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319816.121125:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319816.311788:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319817.014867:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319817.014879:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319817.014883:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319817.014887:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319817.145335:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319817.335187:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319818.169715:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319818.358811:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319819.190849:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319819.382818:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319820.217121:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319820.407532:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319821.239143:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319821.430788:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:2.0:1767319822.137300:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319822.137312:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1767319822.137315:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319822.137319:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319822.263720:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319822.455356:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319823.289861:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319823.479834:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319824.311160:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319824.503321:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319825.347331:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319825.526807:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767319826.230970:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319801/real 1767319801] req@ffff9d1f92f71c00 x1853163904611584/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319826.231015:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f71c00 x1853163904611584/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319826 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319826.231026:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319801/real 1767319801] req@ffff9d1f92f70a80 x1853163904611712/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319826.231036:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f92f70a80 x1853163904611712/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319826 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767319826.231064:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767319826.231100:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767319826.231112:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f71c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904611584:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767319826.231119:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767319826.231125:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767319826.231159:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f92f70a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904611712:192.168.202.140@tcp:38:kworker.0 00000100:00080000:2.0:1767319826.231248:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319826.231254:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767319826.231257:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319826.231263:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 5049 00000100:00080000:2.0:1767319826.231271:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319826.231277:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767319826.231320:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cc700 x1853163904611840/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767319826.231347:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319826.231350:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767319826.231352:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319826.231356:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 5049 00000100:00080000:2.0:1767319826.231366:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319826.231370:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767319826.231380:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767319826.231392:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cce00 x1853163904611968/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767319826.231566:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904611840:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767319826.231718:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904611968:192.168.202.140@tcp:38:kworker.0 00010000:00010000:1.0:1767319826.360266:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319826.551374:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319827.386845:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319827.574802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319828.407041:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319828.598760:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319829.431357:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319829.623305:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319830.455131:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319830.646820:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319831.355560:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319831.355571:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319831.355574:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319831.355577:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319831.478954:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319831.670804:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319832.502847:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319832.695012:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319833.527237:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319833.718937:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319834.553439:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319834.742949:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319835.575281:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319835.766956:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319836.471466:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319836.471477:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319836.471480:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319836.471483:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319836.599296:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319836.790806:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319837.623250:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319837.814923:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319838.648698:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319838.839863:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319839.671209:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319839.862801:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319840.696631:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319840.887306:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:0.0:1767319841.590896:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319841.590907:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1767319841.590911:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319841.590914:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319841.718939:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319841.914491:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319842.743672:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319842.934817:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319843.767913:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319843.965760:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319844.791132:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319844.982823:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319845.815384:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319846.008858:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319846.712061:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319846.712072:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319846.712075:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319846.712078:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319846.839136:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319847.030881:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319847.862889:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319848.054777:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319848.886865:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319849.081975:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319849.911313:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319850.102822:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319850.934906:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319851.126813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:1.0:1767319851.830817:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319826/real 1767319826] req@ffff9d1f967cc700 x1853163904611840/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319851 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319851.830852:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cc700 x1853163904611840/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319851 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319851.830863:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319826/real 1767319826] req@ffff9d1f967cce00 x1853163904611968/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319851 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319851.830874:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f967cce00 x1853163904611968/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319851 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319851.830892:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319851.830901:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319851.830905:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904611840:192.168.202.140@tcp:250:kworker.0 00000100:00080000:1.0:1767319851.830911:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319851.830916:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:1.0:1767319851.830919:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f967cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904611968:192.168.202.140@tcp:38:kworker.0 00000100:00080000:1.0:1767319851.830956:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319851.830960:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1767319851.830963:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767319851.830968:0:470:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 5073 00000100:00080000:1.0:1767319851.830976:0:470:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767319851.831083:0:470:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:1.0:1767319851.831113:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d1880 x1853163904612096/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1767319851.831141:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319851.831144:0:470:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767319851.831146:0:470:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767319851.831149:0:470:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 5073 00000100:00080000:1.0:1767319851.831153:0:470:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767319851.831165:0:470:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:1.0:1767319851.831174:0:470:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767319851.831186:0:470:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f918d3800 x1853163904612224/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767319851.831760:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904612096:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767319851.832063:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904612224:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767319851.964042:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319852.150800:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319852.982972:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319853.174784:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319854.007475:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319854.198824:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319855.031456:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319855.222857:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319856.055184:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319856.248351:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319856.950816:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319856.950826:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319856.950830:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319856.950833:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319857.081605:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319857.270826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319858.107091:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319858.295248:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319859.127110:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319859.319024:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319860.154437:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319860.342826:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319861.179426:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319861.368661:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319862.070788:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319862.070800:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319862.070803:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319862.070806:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1767319862.199007:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319862.390802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319863.223138:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319863.414797:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319864.246945:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319864.438827:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319865.271227:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319865.462815:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319866.296265:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319866.486813:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:3.0:1767319867.193961:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319867.193986:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1767319867.193989:0:470:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319867.193993:0:470:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1767319867.318863:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319867.510831:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319868.342890:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319868.534802:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319869.367427:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319869.564235:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319870.392231:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319870.582830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319871.419364:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319871.606830:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00080000:1.0:1767319872.314248:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319872.314261:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319872.314264:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319872.314267:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1767319872.439634:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319872.630828:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:3.0:1767319873.462993:0:7207:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319873.654921:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:0.0:1767319874.487185:0:15045:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319874.678985:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00010000:00010000:1.0:1767319875.510987:0:15046:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319875.702853:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds 00000100:00100000:3.0:1767319876.406992:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319851/real 1767319851] req@ffff9d1f918d1880 x1853163904612096/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319876 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319876.407125:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d1880 x1853163904612096/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319876 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319876.407136:0:2489:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319851/real 1767319851] req@ffff9d1f918d3800 x1853163904612224/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319876 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319876.407150:0:2489:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d1f918d3800 x1853163904612224/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 1 dl 1767319876 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1767319876.407174:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1767319876.407247:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767319876.407273:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904612096:192.168.202.140@tcp:250:kworker.0 00000100:00080000:3.0:1767319876.407280:0:2489:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319876.407308:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58a3d12-1052-4402-bc3b-a5b3ccbdd169->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319876.407315:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1767319876.407320:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980ea000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319876.407325:0:37:0:(import.c:509:import_select_connection()) MGC192.168.202.140@tcp: connect to NID 192.168.202.140@tcp last attempt 5099 00000100:00080000:2.0:1767319876.407345:0:37:0:(import.c:588:import_select_connection()) MGC192.168.202.140@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319876.407352:0:37:0:(import.c:623:import_select_connection()) MGC192.168.202.140@tcp: import ffff9d1f980ea000 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000100:00100000:2.0:1767319876.407388:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cc000 x1853163904612352/t0(0) o250->MGC192.168.202.140@tcp@192.168.202.140@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1767319876.407415:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45d767c3-7963-411a-be8f-bcbeb1c06075->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1767319876.407418:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1767319876.407420:0:37:0:(import.c:65:import_set_state_nolock()) ffff9d1f980bb800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319876.407423:0:37:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: connect to NID 192.168.202.140@tcp last attempt 5099 00000100:00080000:2.0:1767319876.407427:0:37:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1767319876.407431:0:37:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d1f98126000: import ffff9d1f980bb800 using connection 192.168.202.140@tcp/192.168.202.140@tcp 00000008:00100000:2.0:1767319876.407444:0:37:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767319876.407457:0:37:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d1f967cc380 x1853163904612480/t0(0) o38->lustre-MDT0000-mdc-ffff9d1f98126000@192.168.202.140@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319876.407665:0:2489:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.140@tcp failed (-110) 00000100:00100000:3.0:1767319876.407670:0:2489:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9d1f918d3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904612224:192.168.202.140@tcp:38:kworker.0 00000100:00100000:3.0:1767319876.407720:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58a3d12-1052-4402-bc3b-a5b3ccbdd169:2489:1853163904612352:192.168.202.140@tcp:250:kworker.0 00000100:00100000:3.0:1767319876.407951:0:2489:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9d1f967cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45d767c3-7963-411a-be8f-bcbeb1c06075:2489:1853163904612480:192.168.202.140@tcp:38:kworker.0 00010000:00010000:3.0:1767319876.535940:0:7208:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319876.729281:0:47994:0:(client.c:2690:ptlrpc_set_wait()) set ffff9d1f90b6b9c0 going to sleep for 0 seconds Debug log: 20567 lines, 20567 kept, 0 dropped, 0 bad.