00010000:00010000:2.0F:1767319111.811868:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0F:1767319112.829893:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767319113.872679:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767319114.878264:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0F:1767319115.912643:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1767319116.542744:0:77831:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45c64e87-25b2-41ad-bfa5-81d4dac9e05a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319116.542753:0:77831:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767319116.542758:0:77831:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767319116.542771:0:77831:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: connect to NID 192.168.203.101@tcp last attempt 4315 00000100:00080000:1.0:1767319116.542783:0:77831:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1767319116.542791:0:77831:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: import ffff9001d0032800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1767319116.542805:0:77831:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767319116.542828:0:77831:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9001c8ee3100 x1853164343406080/t0(0) o38->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319116.542871:0:13814:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343406080:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0F:1767319116.543438:0:13815:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:acd6602b-0f35-4bfc-9413-403f1a7558e3:13815:1853164343405952:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319116.543691:0:13818:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:1.0:1767319116.543703:0:13818:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13818:1853164343406336:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319116.543805:0:13818:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13818:1853164343406464:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1767319116.544331:0:13815:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1767319116.544340:0:13815:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13815:1853164343406208:192.168.203.101@tcp:400:kworker.0 00000100:00080000:2.0:1767319116.549926:0:13814:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319116.549961:0:13814:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-19) 00000100:00100000:2.0:1767319116.549968:0:13814:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343406080:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1767319116.557874:0:13815:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13815:1853164343406208:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1767319116.557992:0:13818:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13818:1853164343406464:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1767319116.558007:0:13818:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13818:1853164343406336:192.168.203.101@tcp:400:kworker.0 00010000:00010000:3.0:1767319116.930221:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767319117.952621:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767319118.976058:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767319119.997933:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767319121.023858:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1767319121.661790:0:77831:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45c64e87-25b2-41ad-bfa5-81d4dac9e05a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319121.661799:0:77831:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767319121.661804:0:77831:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767319121.661819:0:77831:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: connect to NID 192.168.203.101@tcp last attempt 4320 00000100:00080000:1.0:1767319121.661835:0:77831:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1767319121.661844:0:77831:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: import ffff9001d0032800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1767319121.661858:0:77831:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319121.661871:0:13815:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:acd6602b-0f35-4bfc-9413-403f1a7558e3:13815:1853164343406592:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319121.661905:0:77831:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9001c8ee0000 x1853164343406720/t0(0) o38->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319121.662009:0:13816:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343406848:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319121.662123:0:13814:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343406720:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1767319121.662193:0:13817:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343406976:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319121.662251:0:13817:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1767319121.662260:0:13817:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343407104:192.168.203.101@tcp:400:kworker.0 00000080:00200000:1.0:1767319121.670222:0:77831:0:(vvp_io.c:1854:vvp_io_init()) [0x200000bd2:0x50:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1767319121.670258:0:77831:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1767319121.670341:0:77831:0:(vvp_io.c:289:vvp_io_fini()) [0x200000bd2:0x50:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1767319121.670356:0:77831:0:(vvp_io.c:1854:vvp_io_init()) [0x240000bd1:0x4e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1767319121.670363:0:77831:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1767319121.670374:0:77831:0:(vvp_io.c:289:vvp_io_fini()) [0x240000bd1:0x4e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1767319121.671366:0:13817:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343406976:192.168.203.101@tcp:400:kworker.0 00000100:00080000:2.0:1767319121.671601:0:13814:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319121.671609:0:13814:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-19) 00000100:00100000:2.0:1767319121.671613:0:13814:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343406720:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1767319121.671779:0:13816:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343406848:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319121.674035:0:13817:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343407104:192.168.203.101@tcp:400:kworker.0 00010000:00010000:2.0:1767319122.047441:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1767319122.709634:0:13817:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319106/real 1767319106] req@ffff9001c8ee2680 x1853164343404544/t0(0) o400->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 224/224 e 0 to 1 dl 1767319122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319122.738695:0:13817:0:(client.c:2522:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9001c8ee2680 x1853164343404544/t0(0) o400->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 224/224 e 0 to 1 dl 1767319122 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767319122.738711:0:13817:0:(import.c:65:import_set_state_nolock()) ffff9001d002e800 MGS: changing import state from FULL to DISCONN 00000100:02020000:0.0:1767319122.738715:0:13817:0:(import.c:185:ptlrpc_set_import_discon()) MGC192.168.203.101@tcp: Connection to MGS (at 192.168.203.101@tcp) was lost; in progress operations using this service will fail 00000100:00080000:0.0:1767319122.756775:0:13817:0:(import.c:191:ptlrpc_set_import_discon()) import MGS@192.168.203.101@tcp for MGC192.168.203.101@tcp not replayable, auto-deactivating 00000100:00080000:0.0:1767319122.756782:0:13817:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:0.0:1767319122.756791:0:13817:0:(client.c:3559:ptlrpc_abort_inflight()) @@@ inflight req@ffff9001c8ee2680 x1853164343404544/t0(0) o400->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 224/224 e 0 to 1 dl 1767319122 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319122.756808:0:13817:0:(client.c:3559:ptlrpc_abort_inflight()) @@@ inflight req@ffff9001c8ee0380 x1853164343405312/t0(0) o400->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 224/224 e 0 to 0 dl 1767319127 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319122.756829:0:13817:0:(client.c:3559:ptlrpc_abort_inflight()) @@@ inflight req@ffff9001c8ee1500 x1853164343405952/t0(0) o400->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 224/224 e 0 to 0 dl 1767319132 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1767319122.756839:0:13817:0:(client.c:3559:ptlrpc_abort_inflight()) @@@ inflight req@ffff9001c8ee0a80 x1853164343406592/t0(0) o400->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 224/224 e 0 to 0 dl 1767319137 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:0.0:1767319122.756848:0:13817:0:(mgc_request.c:1165:mgc_import_event()) import event 0x808001 00000100:00080000:0.0:1767319122.756851:0:13817:0:(pinger.c:192:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1767319122.756852:0:13817:0:(mgc_request.c:1165:mgc_import_event()) import event 0x808002 00000100:00080000:0.0:1767319122.756855:0:13817:0:(import.c:416:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:2.0:1767319122.757102:0:13815:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:acd6602b-0f35-4bfc-9413-403f1a7558e3:13815:1853164343406592:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1767319122.757113:0:13815:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:acd6602b-0f35-4bfc-9413-403f1a7558e3:13815:1853164343405952:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1767319122.757122:0:13815:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:acd6602b-0f35-4bfc-9413-403f1a7558e3:13815:1853164343405312:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1767319122.757171:0:13817:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:acd6602b-0f35-4bfc-9413-403f1a7558e3:13817:1853164343404544:192.168.203.101@tcp:400:kworker.0 00000100:00080000:3.0:1767319122.757599:0:77831:0:(pinger.c:229:ptlrpc_pinger_process_import()) acd6602b-0f35-4bfc-9413-403f1a7558e3->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319122.757604:0:77831:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1767319122.757608:0:77831:0:(import.c:65:import_set_state_nolock()) ffff9001d002e800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319122.757614:0:77831:0:(import.c:509:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 4294 00000100:00080000:3.0:1767319122.757627:0:77831:0:(import.c:623:import_select_connection()) MGC192.168.203.101@tcp: import ffff9001d002e800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1767319122.757657:0:77831:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9001c7703100 x1853164343407232/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1767319122.757678:0:77831:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45c64e87-25b2-41ad-bfa5-81d4dac9e05a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1767319122.757683:0:77831:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1767319122.757684:0:77831:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1767319122.757688:0:77831:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: connect to NID 192.168.203.101@tcp last attempt 4325 00000100:00080000:3.0:1767319122.757692:0:77831:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1767319122.757695:0:77831:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: import ffff9001d0032800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1767319122.757704:0:77831:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1767319122.757715:0:77831:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9001c7700700 x1853164343407360/t0(0) o38->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767319122.757785:0:13818:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:3.0:1767319122.757796:0:13818:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7702680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13818:1853164343407744:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1767319122.758340:0:13814:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7703100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:acd6602b-0f35-4bfc-9413-403f1a7558e3:13814:1853164343407232:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1767319122.758393:0:13814:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7700700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343407360:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1767319122.758475:0:13816:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7700a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343407616:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319122.758509:0:13816:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1767319122.758515:0:13816:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7700380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343407488:192.168.203.101@tcp:400:kworker.0 00000100:00080000:2.0:1767319122.762070:0:13814:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319122.762101:0:13814:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-19) 00000100:00100000:2.0:1767319122.762106:0:13814:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7700700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343407360:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1767319122.762200:0:13816:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7700380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343407488:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319122.762215:0:13816:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7700a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343407616:192.168.203.101@tcp:400:kworker.0 00000100:00100000:3.0:1767319122.762288:0:13818:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7702680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13818:1853164343407744:192.168.203.101@tcp:400:kworker.0 00010000:00010000:1.0:1767319123.070321:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767319124.100127:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767319125.118459:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767319126.141788:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767319127.171275:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1767319127.810085:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) acd6602b-0f35-4bfc-9413-403f1a7558e3->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319127.810095:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) acd6602b-0f35-4bfc-9413-403f1a7558e3->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1767319127.810098:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45c64e87-25b2-41ad-bfa5-81d4dac9e05a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1767319127.810101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1767319127.810105:0:8:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1767319127.810115:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: connect to NID 192.168.203.101@tcp last attempt 4326 00000100:00080000:1.0:1767319127.810128:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1767319127.810134:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: import ffff9001d0032800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1767319127.810144:0:8:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1767319127.810175:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9001c8ee3480 x1853164343407872/t0(0) o38->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1767319127.810282:0:13815:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13815:1853164343408128:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319127.810372:0:13815:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1767319127.810379:0:13815:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13815:1853164343408256:192.168.203.101@tcp:400:kworker.0 00000100:00100000:3.0:1767319127.811456:0:13818:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13818:1853164343408000:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1767319127.813773:0:13814:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343407872:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1767319127.820114:0:13815:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13815:1853164343408256:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319127.820134:0:13815:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13815:1853164343408128:192.168.203.101@tcp:400:kworker.0 00000100:00100000:3.0:1767319127.821317:0:13818:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13818:1853164343408000:192.168.203.101@tcp:400:kworker.0 00000100:00080000:2.0:1767319127.834345:0:13814:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1767319127.834360:0:13814:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-19) 00000100:00100000:2.0:1767319127.834365:0:13814:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343407872:192.168.203.101@tcp:38:kworker.0 00010000:00010000:3.0:1767319128.191480:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767319129.213794:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767319130.239201:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767319131.262506:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767319132.286626:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1767319132.928870:0:13814:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319122/real 1767319122] req@ffff9001c7703100 x1853164343407232/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1767319132.928912:0:13814:0:(client.c:2509:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9001c7703100 x1853164343407232/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1767319132 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319132.928959:0:13814:0:(import.c:65:import_set_state_nolock()) ffff9001d002e800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319132.928971:0:13814:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1767319132.928975:0:13814:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7703100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:acd6602b-0f35-4bfc-9413-403f1a7558e3:13814:1853164343407232:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1767319132.929250:0:77832:0:(pinger.c:229:ptlrpc_pinger_process_import()) acd6602b-0f35-4bfc-9413-403f1a7558e3->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319132.929257:0:77832:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1767319132.929267:0:77832:0:(import.c:65:import_set_state_nolock()) ffff9001d002e800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319132.929273:0:77832:0:(import.c:509:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 4326 00000100:00080000:0.0:1767319132.929285:0:77832:0:(import.c:588:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1767319132.929292:0:77832:0:(import.c:623:import_select_connection()) MGC192.168.203.101@tcp: import ffff9001d002e800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1767319132.929328:0:77832:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9001c76dca80 x1853164343408384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1767319132.929356:0:77832:0:(pinger.c:229:ptlrpc_pinger_process_import()) 45c64e87-25b2-41ad-bfa5-81d4dac9e05a->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1767319132.929360:0:77832:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1767319132.929362:0:77832:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1767319132.929366:0:77832:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: connect to NID 192.168.203.101@tcp last attempt 4332 00000100:00080000:0.0:1767319132.929370:0:77832:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1767319132.929374:0:77832:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: import ffff9001d0032800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1767319132.929386:0:77832:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1767319132.929397:0:77832:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9001c76df480 x1853164343408512/t0(0) o38->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767319132.929513:0:13817:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c76dfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343408640:192.168.203.101@tcp:400:kworker.0 00000100:00100000:0.0:1767319132.929664:0:13817:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:0.0:1767319132.929672:0:13817:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c76ded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343408768:192.168.203.101@tcp:400:kworker.0 00000100:00100000:0.0:1767319132.929713:0:13815:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c76dd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13815:1853164343408896:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319132.930458:0:13814:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c76dca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:acd6602b-0f35-4bfc-9413-403f1a7558e3:13814:1853164343408384:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1767319132.930549:0:13814:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c76df480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343408512:192.168.203.101@tcp:38:kworker.0 00010000:00080000:1.0:1767319132.947316:0:13814:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9001c76df480 x1853164343408512/t0(0) o38->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 520/416 e 0 to 0 dl 1767319157 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1767319132.947344:0:13814:0:(import.c:1123:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff9001d08b3000: connect to target with instance 4 00000100:00080000:1.0:1767319132.947367:0:13814:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1767319132.947372:0:13814:0:(import.c:1679:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1767319132.947375:0:13814:0:(client.c:3081:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9001d08b3000: committing for last_committed 12884901895 gen 1 00000100:00080000:1.0:1767319132.947379:0:13814:0:(recover.c:53:ptlrpc_replay_next()) import ffff9001d0032800 from lustre-MDT0000_UUID committed 12884901895 last 0 00000100:00080000:1.0:1767319132.947383:0:13814:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00010000:00010000:2.0:1767319132.948032:0:13815:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:2.0:1767319132.948046:0:13815:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c76dd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13815:1853164343408896:192.168.203.101@tcp:400:kworker.0 00010000:00010000:1.0:1767319132.948147:0:82079:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1767319132.948172:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767319132.948174:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767319132.948176:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767319132.948452:0:13817:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:0.0:1767319132.948460:0:13817:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c76ded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343408768:192.168.203.101@tcp:400:kworker.0 00010000:00010000:0.0:1767319132.948490:0:13817:0:(ldlm_lockd.c:2188:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 2 blwis in pool 00000100:00100000:0.0:1767319132.948496:0:13817:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c76dfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343408640:192.168.203.101@tcp:400:kworker.0 00010000:00010000:0.0:1767319132.948532:0:82079:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1767319132.948542:0:82079:0:(ldlm_lockd.c:2793:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1767319132.948548:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767319132.948550:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767319132.948571:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767319132.952804:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1767319132.952976:0:13814:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c76df480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343408512:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1767319132.953049:0:13814:0:(import.c:1123:ptlrpc_connect_interpret()) MGC192.168.203.101@tcp: connect to target with instance 0 10000000:01000000:0.0:1767319132.953064:0:13814:0:(mgc_request.c:1165:mgc_import_event()) import event 0x808005 00000100:02000400:0.0:1767319132.953069:0:13814:0:(import.c:1275:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.203.101@tcp) after server handle changed from 0x1c4382d25609f12a to 0x1c4382d25609f441 00010000:00080000:2.0:1767319132.954864:0:83452:0:(ldlm_request.c:2714:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff9001d08b3000 to lustre-MDT0000_UUID@192.168.203.101@tcp 00010000:00010000:2.0:1767319132.954872:0:83452:0:(ldlm_request.c:2634:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff9001d08b3000 (4) 00010000:00010000:2.0:1767319132.954934:0:83452:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c5b5ea00/0xdb4bedfb3a3a84d7 lrc: 0/0,0 mode: --/PR res: [0x240000bd1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x1c4382d25609f1fc expref: -99 pid: 82116 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767319132.954979:0:83452:0:(vvp_object.c:114:vvp_conf_set()) [0x200000bd2:0x50:0x0]: losing layout lock 00010000:00010000:2.0:1767319132.955015:0:83452:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c29cd000/0xdb4bedfb3a3a84c2 lrc: 0/0,0 mode: --/PR res: [0x200000bd2:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x1c4382d25609f1c4 expref: -99 pid: 82115 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1767319132.955045:0:83452:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c29cc000/0xdb4bedfb3a3a84bb lrc: 0/0,0 mode: --/PR res: [0x200000bd2:0x3:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x1c4382d25609f1a8 expref: -99 pid: 82115 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319132.955066:0:83452:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c29cda00/0xdb4bedfb3a3a84b4 lrc: 0/0,0 mode: --/PR res: [0x200000bd2:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x1c4382d25609f18c expref: -99 pid: 82115 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319132.955076:0:83452:0:(ldlm_request.c:2647:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff9001d08b3000 00000100:00080000:2.0:1767319132.955806:0:83452:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00080000:0.0:1767319132.971045:0:13814:0:(import.c:65:import_set_state_nolock()) ffff9001d002e800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:0.0:1767319132.971068:0:13814:0:(import.c:1652:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.203.101@tcp; invalidating 00000100:00100000:3.0:1767319132.986943:0:13814:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c76dca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:acd6602b-0f35-4bfc-9413-403f1a7558e3:13814:1853164343408384:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1767319132.987592:0:13814:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c768a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343409024:192.168.203.101@tcp:400:ldlm_lock_repla.0 00000100:00080000:2.0:1767319132.987743:0:83455:0:(import.c:1584:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.203.101@tcp to MGS@192.168.203.101@tcp 00000100:00100000:2.0:1767319132.987747:0:83455:0:(import.c:321:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1767319132.987752:0:83455:0:(mgc_request.c:1165:mgc_import_event()) import event 0x808003 00010000:00010000:2.0:1767319132.987772:0:83455:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.203.101@tcp lock: ffff9001c29cc200/0xdb4bedfb3a3a843d lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x1c4382d25609f131 expref: -99 pid: 82073 timeout: 0 lvb_type: 0 10000000:00010000:2.0:1767319132.987783:0:83455:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.203.101@tcp lock: ffff9001c29cc200/0xdb4bedfb3a3a843d lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x1c4382d25609f131 expref: -99 pid: 82073 timeout: 0 lvb_type: 0 10000000:01000000:2.0:1767319132.987789:0:83455:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1767319132.987805:0:83455:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1767319132.987815:0:83455:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.203.101@tcp lock: ffff9001c29cc200/0xdb4bedfb3a3a843d lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x1c4382d25609f131 expref: -99 pid: 82073 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319132.987844:0:83455:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.203.101@tcp lock: ffff9001c29ccc00/0xdb4bedfb3a3a8444 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x1c4382d25609f138 expref: -99 pid: 82073 timeout: 0 lvb_type: 0 10000000:00010000:2.0:1767319132.987850:0:83455:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.203.101@tcp lock: ffff9001c29ccc00/0xdb4bedfb3a3a8444 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x1c4382d25609f138 expref: -99 pid: 82073 timeout: 0 lvb_type: 0 10000000:01000000:2.0:1767319132.987856:0:83455:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1767319132.987877:0:83455:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1767319132.987881:0:83455:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.203.101@tcp lock: ffff9001c29ccc00/0xdb4bedfb3a3a8444 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x1c4382d25609f138 expref: -99 pid: 82073 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319132.987903:0:83455:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.203.101@tcp lock: ffff9001c35e7600/0xdb4bedfb3a3a8475 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x1c4382d25609f13f expref: -99 pid: 82073 timeout: 0 lvb_type: 0 10000000:00010000:2.0:1767319132.987909:0:83455:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.203.101@tcp lock: ffff9001c35e7600/0xdb4bedfb3a3a8475 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x1c4382d25609f13f expref: -99 pid: 82073 timeout: 0 lvb_type: 0 10000000:01000000:2.0:1767319132.987935:0:83455:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1767319132.987942:0:83455:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1767319132.987946:0:83455:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.203.101@tcp lock: ffff9001c35e7600/0xdb4bedfb3a3a8475 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x1c4382d25609f13f expref: -99 pid: 82073 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319132.987967:0:83455:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.203.101@tcp lock: ffff9001c6db6200/0xdb4bedfb3a3a847c lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x1c4382d25609f146 expref: -99 pid: 82073 timeout: 0 lvb_type: 0 10000000:00010000:2.0:1767319132.987973:0:83455:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.203.101@tcp lock: ffff9001c6db6200/0xdb4bedfb3a3a847c lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x1c4382d25609f146 expref: -99 pid: 82073 timeout: 0 lvb_type: 0 10000000:01000000:2.0:1767319132.987979:0:83455:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:2.0:1767319132.987984:0:83455:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1767319132.987988:0:83455:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.203.101@tcp lock: ffff9001c6db6200/0xdb4bedfb3a3a847c lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x1c4382d25609f146 expref: -99 pid: 82073 timeout: 0 lvb_type: 0 00000100:00080000:2.0:1767319132.988009:0:83455:0:(import.c:65:import_set_state_nolock()) ffff9001d002e800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:2.0:1767319132.988015:0:83455:0:(import.c:65:import_set_state_nolock()) ffff9001d002e800 MGS: changing import state from RECOVER to FULL 10000000:01000000:2.0:1767319132.988017:0:83455:0:(mgc_request.c:1165:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1767319132.988023:0:83455:0:(pinger.c:185:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:2.0:1767319132.988028:0:83455:0:(import.c:1722:ptlrpc_import_recovery_state_machine()) MGC192.168.203.101@tcp: Connection restored to 192.168.203.101@tcp (at 192.168.203.101@tcp) 00010000:00010000:0.0:1767319133.310010:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767319134.334040:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:3.0:1767319134.910633:0:82082:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1767319134.910643:0:82082:0:(mgc_request.c:1812:mgc_process_log()) Process log lustre-cliir-ffff9001d08b3000 from 11 10000000:01000000:3.0:1767319134.910649:0:82082:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1767319134.910711:0:82082:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.203.101@tcp lock: ffff9001d810dc00/0xdb4bedfb3a3a84fa lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767319134.910722:0:82082:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.203.101@tcp lock: ffff9001d810dc00/0xdb4bedfb3a3a84fa lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767319134.910733:0:82082:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.203.101@tcp lock: ffff9001d810dc00/0xdb4bedfb3a3a84fa lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00000100:00100000:3.0:1767319134.910771:0:82082:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7702680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409152:192.168.203.101@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1767319134.910995:0:82082:0:(client.c:2690:ptlrpc_set_wait()) set ffff9001e056f3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1767319134.922878:0:82082:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7702680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409152:192.168.203.101@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1767319134.922918:0:82082:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.203.101@tcp lock: ffff9001d810dc00/0xdb4bedfb3a3a84fa lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x1c4382d25609f448 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767319134.922932:0:82082:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.203.101@tcp lock: ffff9001d810dc00/0xdb4bedfb3a3a84fa lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x1c4382d25609f448 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00000100:00100000:3.0:1767319134.923044:0:82082:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7702680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409280:192.168.203.101@tcp:256:ll_cfg_requeue.0 00000100:00100000:3.0:1767319134.923165:0:82082:0:(client.c:2690:ptlrpc_set_wait()) set ffff9001e056f3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767319134.930281:0:82082:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7702680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409280:192.168.203.101@tcp:256:ll_cfg_requeue.0 00000020:00000080:2.0:1767319134.930342:0:82082:0:(obd_config.c:1389:class_process_config()) processing cmd: cf00f 10000000:01000000:2.0:1767319134.930499:0:82082:0:(mgc_request.c:1889:mgc_process_log()) MGC192.168.203.101@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:2.0:1767319134.930511:0:82082:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.203.101@tcp lock: ffff9001d810dc00/0xdb4bedfb3a3a84fa lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x1c4382d25609f448 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319134.930525:0:82082:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.203.101@tcp lock: ffff9001d810dc00/0xdb4bedfb3a3a84fa lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x1c4382d25609f448 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 10000000:01000000:2.0:1767319134.930549:0:82082:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:2.0:1767319134.930573:0:82082:0:(mgc_request.c:1812:mgc_process_log()) Process log lustre-client-ffff9001d08b3000 from 48 10000000:01000000:2.0:1767319134.930576:0:82082:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:2.0:1767319134.930622:0:82082:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.203.101@tcp lock: ffff9001c6db3000/0xdb4bedfb3a3a8501 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319134.930630:0:82082:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.203.101@tcp lock: ffff9001c6db3000/0xdb4bedfb3a3a8501 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319134.930640:0:82082:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.203.101@tcp lock: ffff9001c6db3000/0xdb4bedfb3a3a8501 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1767319134.930663:0:82082:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c768a680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409408:192.168.203.101@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1767319134.930758:0:82082:0:(client.c:2690:ptlrpc_set_wait()) set ffff9001c8ad3b40 going to sleep for 11 seconds 00000100:00100000:3.0:1767319134.954426:0:82082:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c768a680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409408:192.168.203.101@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1767319134.954475:0:82082:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.203.101@tcp lock: ffff9001c6db3000/0xdb4bedfb3a3a8501 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x1c4382d25609f44f expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1767319134.954491:0:82082:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.203.101@tcp lock: ffff9001c6db3000/0xdb4bedfb3a3a8501 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x1c4382d25609f44f expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00000100:00100000:3.0:1767319134.954621:0:82082:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7700e00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409536:192.168.203.101@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1767319134.954722:0:82082:0:(client.c:2690:ptlrpc_set_wait()) set ffff9001e056fa80 going to sleep for 11 seconds 00000100:00100000:3.0:1767319134.960594:0:82082:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7700e00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409536:192.168.203.101@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1767319134.960684:0:82082:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7700e00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409664:192.168.203.101@tcp:503:ll_cfg_requeue.0 00000100:00100000:3.0:1767319134.960844:0:82082:0:(client.c:2690:ptlrpc_set_wait()) set ffff9001e056fa80 going to sleep for 11 seconds 00000100:00100000:1.0:1767319134.964901:0:82082:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7700e00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409664:192.168.203.101@tcp:503:ll_cfg_requeue.0 00000040:00080000:3.0:1767319134.975164:0:83489:0:(llog.c:854:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 48 00000020:01000000:1.0:1767319134.990017:0:82082:0:(obd_config.c:2043:class_config_parse_llog()) Processed log lustre-client gen 48-47 (rc=0) 10000000:01000000:1.0:1767319134.990064:0:82082:0:(mgc_request.c:1889:mgc_process_log()) MGC192.168.203.101@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:1.0:1767319134.990078:0:82082:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.203.101@tcp lock: ffff9001c6db3000/0xdb4bedfb3a3a8501 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x1c4382d25609f44f expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767319134.990117:0:82082:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.203.101@tcp lock: ffff9001c6db3000/0xdb4bedfb3a3a8501 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x1c4382d25609f44f expref: -99 pid: 82082 timeout: 0 lvb_type: 0 10000000:01000000:0.0:1767319134.990258:0:82082:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:0.0:1767319134.990262:0:82082:0:(mgc_request.c:1812:mgc_process_log()) Process log params-ffff9001d08b3000 from 7 10000000:01000000:0.0:1767319134.990266:0:82082:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:0.0:1767319134.990322:0:82082:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.203.101@tcp lock: ffff9001cba74600/0xdb4bedfb3a3a8508 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319134.990332:0:82082:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.203.101@tcp lock: ffff9001cba74600/0xdb4bedfb3a3a8508 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319134.990343:0:82082:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.203.101@tcp lock: ffff9001cba74600/0xdb4bedfb3a3a8508 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1767319134.990379:0:82082:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c76df480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409792:192.168.203.101@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1767319134.990674:0:82082:0:(client.c:2690:ptlrpc_set_wait()) set ffff9001c8330840 going to sleep for 11 seconds 00000100:00100000:1.0:1767319134.995363:0:82082:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c76df480 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409792:192.168.203.101@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1767319134.995403:0:82082:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.203.101@tcp lock: ffff9001cba74600/0xdb4bedfb3a3a8508 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x1c4382d25609f456 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767319134.995419:0:82082:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.203.101@tcp lock: ffff9001cba74600/0xdb4bedfb3a3a8508 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x1c4382d25609f456 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00000100:00100000:1.0:1767319134.995537:0:82082:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee3b80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409920:192.168.203.101@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1767319134.995666:0:82082:0:(client.c:2690:ptlrpc_set_wait()) set ffff9001cb898540 going to sleep for 11 seconds 00000100:00100000:1.0:1767319134.997995:0:82082:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee3b80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343409920:192.168.203.101@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1767319134.998071:0:82082:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee3b80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343410048:192.168.203.101@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1767319134.998181:0:82082:0:(client.c:2690:ptlrpc_set_wait()) set ffff9001cb898540 going to sleep for 11 seconds 00000100:00100000:0.0:1767319135.004724:0:82082:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee3b80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343410048:192.168.203.101@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1767319135.007687:0:83490:0:(llog.c:854:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:2.0:1767319135.007903:0:82082:0:(obd_config.c:2043:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:2.0:1767319135.007929:0:82082:0:(mgc_request.c:1889:mgc_process_log()) MGC192.168.203.101@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:2.0:1767319135.007942:0:82082:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.203.101@tcp lock: ffff9001cba74600/0xdb4bedfb3a3a8508 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x1c4382d25609f456 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319135.007955:0:82082:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.203.101@tcp lock: ffff9001cba74600/0xdb4bedfb3a3a8508 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x1c4382d25609f456 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 10000000:01000000:2.0:1767319135.007965:0:82082:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:2.0:1767319135.007968:0:82082:0:(mgc_request.c:1812:mgc_process_log()) Process log lustre-sptlrpc-ffff9001c499e360 from 1 10000000:01000000:2.0:1767319135.007972:0:82082:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:2.0:1767319135.008021:0:82082:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.203.101@tcp lock: ffff9001c6db3200/0xdb4bedfb3a3a850f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319135.008031:0:82082:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.203.101@tcp lock: ffff9001c6db3200/0xdb4bedfb3a3a850f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319135.008064:0:82082:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.203.101@tcp lock: ffff9001c6db3200/0xdb4bedfb3a3a850f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1767319135.008098:0:82082:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c768ad80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343410176:192.168.203.101@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1767319135.008206:0:82082:0:(client.c:2690:ptlrpc_set_wait()) set ffff9001c8ad30c0 going to sleep for 11 seconds 00000100:00100000:1.0:1767319135.032600:0:82082:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c768ad80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343410176:192.168.203.101@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1767319135.032639:0:82082:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.203.101@tcp lock: ffff9001c6db3200/0xdb4bedfb3a3a850f lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x1c4382d25609f45d expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1767319135.032655:0:82082:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.203.101@tcp lock: ffff9001c6db3200/0xdb4bedfb3a3a850f lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x1c4382d25609f45d expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1767319135.032852:0:82082:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c768b100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343410304:192.168.203.101@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1767319135.032946:0:82082:0:(client.c:2690:ptlrpc_set_wait()) set ffff9001c8ad33c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767319135.041138:0:82082:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c768b100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:acd6602b-0f35-4bfc-9413-403f1a7558e3:82082:1853164343410304:192.168.203.101@tcp:501:ll_cfg_requeue.0 10000000:01000000:0.0:1767319135.043332:0:82082:0:(mgc_request.c:1889:mgc_process_log()) MGC192.168.203.101@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:0.0:1767319135.043374:0:82082:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.203.101@tcp lock: ffff9001c6db3200/0xdb4bedfb3a3a850f lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x1c4382d25609f45d expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319135.043388:0:82082:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.203.101@tcp lock: ffff9001c6db3200/0xdb4bedfb3a3a850f lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x1c4382d25609f45d expref: -99 pid: 82082 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319135.358055:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767319136.390534:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:0.0:1767319137.006738:0:13814:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9001c768a300 x1853164343409024/t0(0) o400->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 224/224 e 0 to 0 dl 1767319148 ref 1 fl Rpc:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767319137.006775:0:13814:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:0.0:1767319137.006783:0:13814:0:(client.c:3203:ptlrpc_resend_req()) @@@ going to resend req@ffff9001c768a300 x1853164343409024/t0(0) o400->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 224/224 e 0 to 0 dl 1767319148 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767319137.006792:0:13814:0:(client.c:3212:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff9001c768a300 x1853164343409024/t0(0) o400->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 224/224 e 0 to 0 dl 1767319148 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1767319137.006811:0:13814:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:2.0:1767319137.008034:0:13814:0:(import.c:1722:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff9001d08b3000: Connection restored to 192.168.203.101@tcp (at 192.168.203.101@tcp) 00000100:00100000:2.0:1767319137.031013:0:13814:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c768a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343409024:192.168.203.101@tcp:400:ldlm_lock_repla.0 00010000:00010000:3.0:1767319137.412294:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1767319138.054067:0:13816:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:acd6602b-0f35-4bfc-9413-403f1a7558e3:13816:1853164343410432:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319138.054638:0:13818:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13818:1853164343410688:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319138.054817:0:13818:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:1.0:1767319138.054826:0:13818:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13818:1853164343410560:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319138.054866:0:13816:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343410944:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1767319138.054892:0:13816:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1767319138.054898:0:13816:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c8ee2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343410816:192.168.203.101@tcp:400:kworker.0 00000100:00100000:3.0:1767319138.059322:0:13816:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:acd6602b-0f35-4bfc-9413-403f1a7558e3:13816:1853164343410432:192.168.203.101@tcp:400:kworker.0 00010000:00080000:3.0:1767319138.061259:0:13818:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9001c8ee3b80 x1853164343410560/t0(0) o400->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 224/224 e 0 to 0 dl 1767319154 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1767319138.061284:0:13818:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13818:1853164343410560:192.168.203.101@tcp:400:kworker.0 00000100:00100000:3.0:1767319138.061298:0:13818:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13818:1853164343410688:192.168.203.101@tcp:400:kworker.0 00000100:00100000:3.0:1767319138.065392:0:13816:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343410816:192.168.203.101@tcp:400:kworker.0 00000100:00100000:3.0:1767319138.065418:0:13816:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c8ee2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343410944:192.168.203.101@tcp:400:kworker.0 00010000:00010000:2.0:1767319138.442663:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767319139.470760:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1767319140.225329:0:83659:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-scrub test 17b: ENOSPC on .. insertion shouldn't leak inodes ========================================================== 20:58:58 (1767319138) 00010000:00010000:2.0:1767319140.480012:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1767319141.225906:0:83680:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9001d1239148),name=/ 00000002:00010000:3.0:1767319141.226054:0:83680:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1767319141.226297:0:83680:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 83680 initiator: MDT0 00010000:00010000:3.0:1767319141.226310:0:83680:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 83680 initiator: MDT0 00010000:00010000:3.0:1767319141.226322:0:83680:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 83680 initiator: MDT0 00000100:00100000:3.0:1767319141.226355:0:83680:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7701180 pname:cluuid:pid:xid:nid:opc:job lfs:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:83680:1853164343411072:192.168.203.101@tcp:101:lfs.0 00000100:00100000:3.0:1767319141.226588:0:83680:0:(client.c:2690:ptlrpc_set_wait()) set ffff9001d01d46c0 going to sleep for 11 seconds 00000100:00100000:2.0:1767319141.266405:0:83680:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7701180 pname:cluuid:pid:xid:nid:opc:job lfs:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:83680:1853164343411072:192.168.203.101@tcp:101:lfs.0 00010000:00010000:2.0:1767319141.266456:0:83680:0:(ldlm_request.c:702:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319141.266471:0:83680:0:(ldlm_request.c:730:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319141.266486:0:83680:0:(ldlm_lock.c:1065:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319141.266520:0:83680:0:(ldlm_request.c:784:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319141.266548:0:83680:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319141.266615:0:83680:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319141.266623:0:83680:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1767319141.266637:0:83680:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9001c7701180 x1853164343411072/t0(0) o101->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 576/688 e 0 to 0 dl 1767319157 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1767319141.266652:0:83680:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319141.266667:0:83680:0:(ldlm_lock.c:1532:ldlm_lock_match_with_skip()) ### not matched ns ffff9001c4c17800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1767319141.266675:0:83680:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1767319141.266706:0:83680:0:(llite_lib.c:2981:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1767319141.266713:0:83680:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9001d1239148) for lock 0xdb4bedfb3a3a8516 00000080:00010000:2.0:1767319141.266719:0:83680:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9001d1239148) 00000080:00010000:2.0:1767319141.266723:0:83680:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xdb4bedfb3a3a8516 from it ffffb60207e73b58 00010000:00010000:2.0:1767319141.266726:0:83680:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1767319141.266733:0:83680:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1767319141.266769:0:83680:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9001d1239148), inode mode 41ed mask 1 00000080:00200000:2.0:1767319141.267257:0:83680:0:(namei.c:1277:ll_lookup_nd()) VFS Op:name=d17b.sanity-scrub, dir=[0x200000007:0x1:0x0](ffff9001d1239148), flags=13 00000080:00200000:2.0:1767319141.267262:0:83680:0:(namei.c:1020:ll_lookup_it()) VFS Op:name=d17b.sanity-scrub, dir=[0x200000007:0x1:0x0](ffff9001d1239148), intent=getattr 00000002:00010000:2.0:1767319141.267293:0:83680:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d17b.sanity-scrub,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1767319141.267352:0:83680:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c6db2800/0xdb4bedfb3a3a851d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 83680 initiator: MDT0 00010000:00010000:2.0:1767319141.267360:0:83680:0:(ldlm_request.c:1022:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c6db2800/0xdb4bedfb3a3a851d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 83680 initiator: MDT0 00010000:00010000:2.0:1767319141.267371:0:83680:0:(ldlm_request.c:1109:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c6db2800/0xdb4bedfb3a3a851d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 83680 initiator: MDT0 00000100:00100000:2.0:1767319141.267398:0:83680:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c768a300 pname:cluuid:pid:xid:nid:opc:job lfs:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:83680:1853164343411200:192.168.203.101@tcp:101:lfs.0 00000100:00100000:2.0:1767319141.267523:0:83680:0:(client.c:2690:ptlrpc_set_wait()) set ffff9001c8ad3600 going to sleep for 11 seconds 00000100:00100000:0.0:1767319141.271024:0:83680:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c768a300 pname:cluuid:pid:xid:nid:opc:job lfs:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:83680:1853164343411200:192.168.203.101@tcp:101:lfs.0 00010000:00010000:0.0:1767319141.271054:0:83680:0:(ldlm_request.c:631:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c6db2800/0xdb4bedfb3a3a851d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 83680 initiator: MDT0 00010000:00010000:0.0:1767319141.271069:0:83680:0:(ldlm_request.c:548:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c6db2800/0xdb4bedfb3a3a851d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 83680 initiator: MDT0 00010000:00010000:0.0:1767319141.271087:0:83680:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c6db2800/0xdb4bedfb3a3a851d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 83680 initiator: MDT0 00010000:00010000:0.0:1767319141.271095:0:83680:0:(ldlm_lock.c:880:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c6db2800/0xdb4bedfb3a3a851d lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 83680 initiator: MDT0 00010000:00010000:0.0:1767319141.271122:0:83680:0:(ldlm_lockd.c:1906:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c6db2800/0xdb4bedfb3a3a851d lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 83680 initiator: MDT0 00010000:00010000:0.0:1767319141.271128:0:83680:0:(ldlm_lockd.c:1920:ldlm_handle_bl_callback()) Lock ffff9001c6db2800 already unused, calling callback (ffffffffc1913b40) 00010000:00010000:0.0:1767319141.271133:0:83680:0:(ldlm_request.c:1595:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c6db2800/0xdb4bedfb3a3a851d lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 83680 initiator: MDT0 00010000:00010000:0.0:1767319141.271140:0:83680:0:(ldlm_request.c:1605:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c6db2800/0xdb4bedfb3a3a851d lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 83680 initiator: MDT0 00010000:00010000:0.0:1767319141.271148:0:83680:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c6db2800/0xdb4bedfb3a3a851d lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 83680 initiator: MDT0 00010000:00010000:0.0:1767319141.271156:0:83680:0:(ldlm_request.c:1301:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1767319141.271164:0:83680:0:(ldlm_lockd.c:1932:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c6db2800/0xdb4bedfb3a3a851d lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 83680 initiator: MDT0 00010000:00010000:0.0:1767319141.271171:0:83680:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001c6db2800/0xdb4bedfb3a3a851d lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 83680 initiator: MDT0 00000002:00100000:0.0:1767319141.271194:0:83680:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9001c768a300 x1853164343411200/t0(0) o101->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 592/584 e 0 to 0 dl 1767319157 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1767319141.271210:0:83680:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1767319141.271247:0:83680:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319141.271263:0:83680:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00000080:00010000:0.0:1767319141.271273:0:83680:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xdb4bedfb3a3a8516 from it ffffb60207e73940 00010000:00010000:0.0:1767319141.271276:0:83680:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319141.271291:0:83680:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767319141.271390:0:83680:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9001d1239148),name=/ 00000002:00010000:0.0:1767319141.271407:0:83680:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767319141.271414:0:83680:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319141.271433:0:83680:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767319141.271439:0:83680:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767319141.271444:0:83680:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9001d1239148) 00000080:00010000:0.0:1767319141.271447:0:83680:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9001d1239148) for lock 0xdb4bedfb3a3a8516 00000080:00010000:0.0:1767319141.271452:0:83680:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xdb4bedfb3a3a8516 from it ffffb60207e73be8 00010000:00010000:0.0:1767319141.271455:0:83680:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319141.271463:0:83680:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767319141.271471:0:83680:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9001d1239148), inode mode 41ed mask 1 00000080:00200000:0.0:1767319141.271479:0:83680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d17b.sanity-scrub, flags=16392 00000080:00200008:0.0:1767319141.271508:0:83680:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9001d1239148), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1767319141.271513:0:83680:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9001d1239148),name=/ 00000002:00010000:0.0:1767319141.271521:0:83680:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767319141.271527:0:83680:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319141.271541:0:83680:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767319141.271548:0:83680:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767319141.271610:0:83680:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9001d1239148) 00000080:00010000:0.0:1767319141.271613:0:83680:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9001d1239148) for lock 0xdb4bedfb3a3a8516 00000080:00010000:0.0:1767319141.271617:0:83680:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xdb4bedfb3a3a8516 from it ffffb60207e73cb8 00010000:00010000:0.0:1767319141.271620:0:83680:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319141.271627:0:83680:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00000080:00200008:0.0:1767319141.272246:0:83680:0:(file.c:6252:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9001d1239148), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1767319141.272253:0:83680:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9001d1239148),name=/ 00000002:00010000:0.0:1767319141.272264:0:83680:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767319141.272274:0:83680:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319141.272283:0:83680:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767319141.272289:0:83680:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767319141.272294:0:83680:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9001d1239148) 00000080:00010000:0.0:1767319141.272297:0:83680:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9001d1239148) for lock 0xdb4bedfb3a3a8516 00000080:00010000:0.0:1767319141.272301:0:83680:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xdb4bedfb3a3a8516 from it ffffb60207e73cb8 00010000:00010000:0.0:1767319141.272305:0:83680:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319141.272318:0:83680:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767319141.272473:0:83680:0:(file.c:6147:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9001d1239148),name=/ 00000002:00010000:0.0:1767319141.272484:0:83680:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1767319141.272491:0:83680:0:(ldlm_lock.c:768:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319141.272499:0:83680:0:(ldlm_lock.c:1515:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767319141.272505:0:83680:0:(namei.c:955:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1767319141.272509:0:83680:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9001d1239148) 00000080:00010000:0.0:1767319141.272512:0:83680:0:(llite_internal.h:2004:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9001d1239148) for lock 0xdb4bedfb3a3a8516 00000080:00010000:0.0:1767319141.272515:0:83680:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xdb4bedfb3a3a8516 from it ffffb60207e73bb0 00010000:00010000:0.0:1767319141.272519:0:83680:0:(ldlm_lock.c:822:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319141.272527:0:83680:0:(ldlm_lock.c:905:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1767319141.272536:0:83680:0:(file.c:6629:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9001d1239148), inode mode 41ed mask 44 00000080:00200008:0.0:1767319141.272543:0:83680:0:(file.c:1001:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9001d1239148), flags 300000 00000080:00200080:0.0:1767319141.272598:0:83680:0:(dir.c:2017:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9001d1239148) cmd=c03066f0 arg=7ffc6dba4500 00000080:00200000:0.0:1767319141.272613:0:83680:0:(dir.c:483:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9001d1239148) name=d17b.sanity-scrub stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:0.0:1767319141.272642:0:83680:0:(ldlm_request.c:1285:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319141.272724:0:83680:0:(ldlm_request.c:1332:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1767319141.272731:0:83680:0:(ldlm_request.c:1378:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1767319141.272734:0:83680:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9001d08b3000 lock: ffff9001d810d400/0xdb4bedfb3a3a8516 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x1c4382d25609f647 expref: -99 pid: 83680 timeout: 0 lvb_type: 0 00000020:00100000:0.0:1767319141.272762:0:83680:0:(genops.c:2438:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9001d08b3000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1767319141.272775:0:83680:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9001c76dfb80 x1853164343411456/t0(0) o36->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1767319141.272804:0:83680:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c76dfb80 pname:cluuid:pid:xid:nid:opc:job lfs:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:83680:1853164343411456:192.168.203.101@tcp:36:lfs.0 00000100:00100000:0.0:1767319141.284858:0:83680:0:(client.c:2690:ptlrpc_set_wait()) set ffff9001cb898c00 going to sleep for 11 seconds 00000100:00100000:0.0:1767319141.302344:0:83680:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c76dfb80 pname:cluuid:pid:xid:nid:opc:job lfs:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:83680:1853164343411456:192.168.203.101@tcp:36:lfs.0 00000080:00200008:0.0:1767319141.302740:0:83680:0:(file.c:424:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9001d1239148), flags 300000 00010000:00010000:2.0:1767319141.501979:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767319142.527976:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1767319143.166157:0:13816:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7702300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343411968:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1767319143.166251:0:13817:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7701500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:acd6602b-0f35-4bfc-9413-403f1a7558e3:13817:1853164343411584:192.168.203.101@tcp:400:kworker.0 00000100:00100000:3.0:1767319143.166287:0:13816:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1767319143.166294:0:13817:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7701880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343412096:192.168.203.101@tcp:400:kworker.0 00000100:00100000:3.0:1767319143.166295:0:13816:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7703b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343411840:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1767319143.166315:0:13817:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:2.0:1767319143.166320:0:13817:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c7700000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343411712:192.168.203.101@tcp:400:kworker.0 00000100:00100000:0.0:1767319143.172224:0:13817:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7700000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343411712:192.168.203.101@tcp:400:kworker.0 00000100:00100000:0.0:1767319143.172247:0:13817:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7701500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:acd6602b-0f35-4bfc-9413-403f1a7558e3:13817:1853164343411584:192.168.203.101@tcp:400:kworker.0 00000100:00100000:0.0:1767319143.172260:0:13817:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7701880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343412096:192.168.203.101@tcp:400:kworker.0 00000100:00100000:3.0:1767319143.172611:0:13816:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7703b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343411840:192.168.203.101@tcp:400:kworker.0 00000100:00100000:3.0:1767319143.172625:0:13816:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c7702300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13816:1853164343411968:192.168.203.101@tcp:400:kworker.0 00010000:00010000:1.0:1767319143.550024:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1767319144.576352:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767319145.597922:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767319146.622026:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1767319147.646213:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1767319148.286057:0:13817:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c76dc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343412608:192.168.203.101@tcp:400:kworker.0 00000100:00100000:0.0:1767319148.286167:0:13817:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [3->2] 00000100:00100000:0.0:1767319148.286175:0:13817:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c76dfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:acd6602b-0f35-4bfc-9413-403f1a7558e3:13817:1853164343412224:192.168.203.101@tcp:400:kworker.0 00000100:00100000:0.0:1767319148.286205:0:13817:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c76de680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343412736:192.168.203.101@tcp:400:kworker.0 00000100:00100000:0.0:1767319148.286254:0:13815:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c76ded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13815:1853164343412352:192.168.203.101@tcp:400:kworker.0 00000100:00100000:0.0:1767319148.286279:0:13815:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1767319148.286285:0:13815:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c76dd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13815:1853164343412480:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1767319148.297092:0:13815:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c76dd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13815:1853164343412480:192.168.203.101@tcp:400:kworker.0 00000100:00080000:2.0:1767319148.297111:0:13815:0:(recover.c:203:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff9001d08b3000 of lustre-MDT0000_UUID@192.168.203.101@tcp abruptly disconnected: reconnecting 00000100:00080000:2.0:1767319148.297117:0:13815:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1767319148.297121:0:13815:0:(import.c:181:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff9001d08b3000: Connection to lustre-MDT0000 (at 192.168.203.101@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00100000:0.0:1767319148.297905:0:13817:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c76de680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343412736:192.168.203.101@tcp:400:kworker.0 00000100:00100000:0.0:1767319148.297923:0:13817:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c76dc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13817:1853164343412608:192.168.203.101@tcp:400:kworker.0 00000100:00080000:2.0:1767319148.311984:0:13815:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1767319148.312003:0:13815:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: connect to NID 192.168.203.101@tcp last attempt 4337 00000100:00080000:2.0:1767319148.312020:0:13815:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9001d08b3000: import ffff9001d0032800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1767319148.312033:0:13815:0:(osc_request.c:3784:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34668544 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1767319148.312066:0:13815:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9001c768b480 x1853164343412864/t0(0) o38->lustre-MDT0000-mdc-ffff9001d08b3000@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1767319148.313118:0:13814:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@ffff9001c768b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343412864:192.168.203.101@tcp:38:ptlrpcd_00_00.0 00000100:00100000:2.0:1767319148.326387:0:13815:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c76ded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13815:1853164343412352:192.168.203.101@tcp:400:kworker.0 00000100:00080000:1.0:1767319148.327273:0:13814:0:(import.c:65:import_set_state_nolock()) ffff9001d0032800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1767319148.327284:0:13814:0:(import.c:1499:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-19) 00000100:00100000:1.0:1767319148.327288:0:13814:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@ffff9001c768b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:45c64e87-25b2-41ad-bfa5-81d4dac9e05a:13814:1853164343412864:192.168.203.101@tcp:38:ptlrpcd_00_00.0 00010000:00010000:0.0:1767319148.670416:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1767319149.694843:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767319150.717837:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1767319150.925273:0:83880:0:(debug.c:665:libcfs_debug_mark_buffer()) DEBUG MARKER: sanity-scrub test_17b: @@@@@@ FAIL: e2fsck -d -v -t -t -f -n /dev/vdc -m8 returned 4, should be <= 0 00010000:00010000:0.0:1767319151.745161:0:82079:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1767319152.766763:0:82080:0:(ldlm_lockd.c:2796:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 496 lines, 496 kept, 0 dropped, 0 bad.